Oct 30 00:37:59.555519 ip-10-0-130-5 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:37:59.918733 ip-10-0-130-5 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:37:59.918733 ip-10-0-130-5 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:37:59.918733 ip-10-0-130-5 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:37:59.918733 ip-10-0-130-5 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:37:59.922336 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922254    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:37:59.922336 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922336    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:37:59.922336 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922339    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922343    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922346    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922349    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922352    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922355    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922358    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922360    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922364    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922367    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922369    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922374    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922378    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922382    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922385    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922388    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922390    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922393    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922396    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:37:59.922424 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922399    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922401    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922404    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922406    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922409    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922423    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922426    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922429    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922432    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922435    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922438    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922440    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922443    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922446    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922448    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922451    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922453    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922456    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922458    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922461    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:37:59.922869 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922463    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922466    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922468    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922471    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922473    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922476    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922478    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922481    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922484    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922486    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922489    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922491    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922495    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922498    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922501    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922504    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922507    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922509    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922518    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:37:59.923352 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922521    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922523    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922526    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922528    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922531    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922533    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922536    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922539    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922541    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922544    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922546    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922549    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922551    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922554    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922556    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922559    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922561    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922564    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922566    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922569    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:37:59.923794 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922571    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922574    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922576    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922579    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922582    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922585    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922587    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922591    2567 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922593    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922602    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922604    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922607    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922615    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922618    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922620    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922623    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922625    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922635    2567 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922638    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922640    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:37:59.924279 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922643    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922645    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922648    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922651    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922653    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.922658    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923454    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923462    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923465    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923467    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923470    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923473    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923476    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923478    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923482    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923485    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923488    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923491    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923495    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:37:59.924735 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923500    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923503    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923506    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923508    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923511    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923514    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923517    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923520    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923523    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923525    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923528    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923531    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923533    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923536    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923538    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923541    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923544    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923546    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923549    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923551    2567 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:37:59.925196 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923554    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923556    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923559    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923561    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923565    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923568    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923570    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923574    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923577    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923580    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923582    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923586    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923588    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923591    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923593    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923596    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923598    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923601    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923603    2567 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923612    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:37:59.925681 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923615    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923617    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923620    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923624    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923627    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923636    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923639    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923642    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923645    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923647    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923650    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923653    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923655    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923658    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923660    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923663    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923666    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923669    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923671    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:37:59.926156 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923674    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923677    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923679    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923681    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923684    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923687    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923690    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923692    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923695    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923697    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923700    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923702    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923705    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923708    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923711    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923713    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923715    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923718    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923721    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:37:59.926625 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923723    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923725    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923728    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923731    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923733    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923736    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923738    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923741    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923743    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923746    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923749    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923751    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923754    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923757    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.923760    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924264    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924270    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924273    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924276    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:37:59.927071 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924279    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924282    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924285    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924287    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924290    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924293    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924296    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924299    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924302    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924305    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924307    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924310    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924313    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924317    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924321    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924323    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924326    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924328    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924331    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:37:59.927532 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924333    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924336    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924339    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924341    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924344    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924347    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924351    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924354    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924358    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924360    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924363    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924366    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924369    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924372    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924375    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924377    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924380    2567 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924382    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924385    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:37:59.928001 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924388    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924390    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924393    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924396    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924398    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924401    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924403    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924406    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924408    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924411    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924413    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924416    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924418    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924421    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924423    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924426    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924428    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924431    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924433    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924436    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:37:59.928470 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924441    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924444    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924447    2567 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924449    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924452    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924455    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924458    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924461    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924463    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924466    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924468    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924471    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924473    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924476    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924479    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924482    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924484    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924487    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924489    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924492    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:37:59.928940 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924494    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924497    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924499    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924501    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924504    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924507    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924509    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924512    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924514    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924517    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924519    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924522    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924525    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924530    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924532    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924535    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924537    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924540    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924542    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924545    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:37:59.929410 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924547    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924550    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924552    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.924556    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925635    2567 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925645    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925652    2567 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925657    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925663    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925666    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925670    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925674    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925677    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925680    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925684    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925687    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925691    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925693    2567 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925697    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925700    2567 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925702    2567 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925705    2567 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925710    2567 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925713    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:37:59.929907 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925717    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925720    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925724    2567 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925728    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925731    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925734    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925737    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925740    2567 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925743    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925746    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925749    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925751    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925756    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925758    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925761    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925764    2567 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925767    2567 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925771    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925775    2567 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925778    2567 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925781    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925784    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925787    2567 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925792    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:37:59.930479 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925794    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925797    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925800    2567 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925803    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925806    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925810    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925813    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925816    2567 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925819    2567 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925822    2567 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925826    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925830    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925833    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925836    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925839    2567 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925842    2567 flags.go:64] FLAG: --help="false"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925845    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-130-5.ec2.internal"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925848    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925851    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925854    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925857    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925860    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925863    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:37:59.931162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925866    2567 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925869    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925872    2567 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925875    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925879    2567 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925882    2567 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925884    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925887    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925890    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925893    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925896    2567 flags.go:64] FLAG: --lock-file=""
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925899    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925902    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925905    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925911    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925914    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925917    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925919    2567 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925923    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925926    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925929    2567 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925932    2567 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925936    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925939    2567 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925943    2567 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:37:59.931759 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925946    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925949    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925952    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925955    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925958    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925961    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925964    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925971    2567 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925974    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925976    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925979    2567 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925983    2567 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925986    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925989    2567 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925992    2567 flags.go:64] FLAG: --port="10250"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925994    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.925997    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-062ddcd3c54899bb9"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926001    2567 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926004    2567 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926007    2567 flags.go:64] FLAG: --register-node="true"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926009    2567 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926013    2567 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926016    2567 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926019    2567 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926022    2567 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926026    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:37:59.932404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926029    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926033    2567 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926036    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926038    2567 flags.go:64] FLAG: --runonce="false"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926041    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926044    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926047    2567 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926050    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926053    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926056    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926058    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926062    2567 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926065    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926068    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926070    2567 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926073    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926076    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926079    2567 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926083    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926088    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926091    2567 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926094    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926098    2567 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926102    2567 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926105    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:37:59.933010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926108    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926111    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926114    2567 flags.go:64] FLAG: --v="2"
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926119    2567 flags.go:64] FLAG: --version="false"
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926122    2567 flags.go:64] FLAG: --vmodule=""
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926126    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926131    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926279    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926284    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926287    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926289    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926292    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926295    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926297    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926300    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926303    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926305    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926308    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926310    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926314    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926319    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:37:59.933599 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926322    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926325    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926328    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926330    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926333    2567 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926336    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926339    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926341    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926344    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926348    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926352    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926355    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926358    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926360    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926363    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926365    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926368    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926372    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926375    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926377    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:37:59.934094 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926380    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926382    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926385    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926388    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926390    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926393    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926395    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926398    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926400    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926403    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926406    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926408    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926410    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926413    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926416    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926418    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926421    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926423    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926430    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926433    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:37:59.934585 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926435    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926439    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926442    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926444    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926447    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926450    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926452    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926455    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926458    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926461    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926464    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926466    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926469    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926471    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926474    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926477    2567 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926480    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926482    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926485    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926488    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:37:59.935089 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926490    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926492    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926495    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926497    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926500    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926502    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926505    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926508    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926510    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926513    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926515    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926519    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926521    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926525    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926528    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926530    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926533    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926536    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926539    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:37:59.935683 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926544    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926547    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926551    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926553    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926556    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926558    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926561    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926564    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926566    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926569    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926571    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926574    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926576    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.926626    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926662    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:37:59.936443 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926667    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926670    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926674    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926676    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926679    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926681    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926684    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926686    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926689    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926691    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926695    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926699    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926704    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926707    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926710    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926712    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926715    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926718    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926720    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:37:59.936842 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926724    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926727    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926729    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926732    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926734    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926737    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926739    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926742    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926744    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926747    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926749    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926752    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926754    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926757    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926759    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926762    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926764    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926767    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926771    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:37:59.937319 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926774    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926777    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926780    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926783    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926785    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926788    2567 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926792    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926795    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926797    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926799    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926802    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926806    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926809    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926813    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926815    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926818    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926821    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926823    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926826    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926829    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:37:59.937782 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926831    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926834    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926837    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926839    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926842    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926844    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926847    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926849    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926852    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926855    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926857    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926860    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926862    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926864    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926867    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926870    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926872    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926875    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926879    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926881    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:37:59.938337 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926884    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926887    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926889    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926892    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926894    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926899    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926902    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926905    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926907    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926910    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926912    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926915    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926917    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926920    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926923    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926926    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926929    2567 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926931    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926934    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926936    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:37:59.938963 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926939    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926941    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926944    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926946    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926948    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926951    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.926953    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.934365    2567 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.934382    2567 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.934401    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.934405    2567 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934462    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934468    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934471    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934474    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934477    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934480    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934482    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934485    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934488    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:37:59.939455 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934491    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934493    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934496    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934498    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934501    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934504    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934507    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934509    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934513    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934515    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934518    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934521    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934524    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934526    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934529    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934531    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934534    2567 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934536    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934539    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934542    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:37:59.939936 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934545    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934548    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934551    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934554    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934556    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934559    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934561    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934564    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934566    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934569    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934571    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934574    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934577    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934579    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934582    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934584    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934587    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934591    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934595    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:37:59.940411 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934599    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934604    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934607    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934610    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934612    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934615    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934617    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934620    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934623    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934626    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934628    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934631    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934633    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934636    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934639    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934642    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934645    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934648    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934650    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:37:59.940880 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934653    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934656    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934658    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934660    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934663    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934666    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934668    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934671    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934673    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934676    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934679    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934681    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934684    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934686    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934689    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934692    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934695    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934697    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934700    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:37:59.941394 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934702    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934705    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934708    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934710    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934713    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934715    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934718    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934720    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934722    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934725    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934728    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934731    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934733    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934736    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934739    2567 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934741    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934744    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934746    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934750    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:37:59.941937 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934754    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.934782    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934926    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934931    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934934    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934937    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934939    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934942    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934944    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934947    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934949    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934952    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934955    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934957    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934960    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934962    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:37:59.942409 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934965    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934968    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934972    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934975    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934977    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934980    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934982    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934990    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934992    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934995    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.934997    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935000    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935003    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935005    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935008    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935010    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935013    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935016    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935018    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935021    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:37:59.942828 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935023    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935026    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935029    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935031    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935034    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935037    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935040    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935043    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935046    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935049    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935051    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935054    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935056    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935059    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935061    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935064    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935067    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935069    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935071    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:37:59.943451 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935074    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935077    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935079    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935082    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935084    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935087    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935089    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935092    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935094    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935097    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935099    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935102    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935104    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935107    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935109    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935112    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935114    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935117    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935119    2567 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:37:59.943946 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935122    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935124    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935127    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935130    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935132    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935135    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935137    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935140    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935142    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935144    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935147    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935149    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935152    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935154    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935156    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935162    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935166    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935170    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935173    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:37:59.944413 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935176    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935178    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935181    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935183    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935186    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935189    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935191    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935193    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935196    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935199    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935201    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935204    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935206    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935209    2567 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935211    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.935251    2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935284    2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:37:59.945159 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935288    2567 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935290    2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935294    2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935296    2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935299    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935301    2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935304    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935307    2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935309    2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935312    2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935314    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935317    2567 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935359    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.935405    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938274    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938320    2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938324    2567 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938460    2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938464    2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938467    2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:37:59.945945 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938470    2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938473    2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938476    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938479    2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938482    2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938485    2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938488    2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938491    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938495    2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938500    2567 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938503    2567 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938506    2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938509    2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938512    2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938514    2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938517    2567 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938524    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938526    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938530    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938532    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:37:59.946770 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938535    2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938538    2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938541    2567 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938543    2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938549    2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938552    2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938554    2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938557    2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938560    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938563    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938565    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938568    2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938571    2567 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938574    2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938576    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938579    2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938582    2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938584    2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938587    2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:37:59.947420 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938590    2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938592    2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938595    2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938598    2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938600    2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938603    2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938606    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938608    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938611    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938616    2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938618    2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938621    2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938623    2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938626    2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938628    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938631    2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938633    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938636    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938638    2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:37:59.947878 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938641    2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938644    2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938646    2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938649    2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938651    2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938654    2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938657    2567 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938661    2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938663    2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938665    2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938668    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938671    2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938673    2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938676    2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938678    2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938681    2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938683    2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938686    2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938689    2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:37:59.948377 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938692    2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938694    2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938697    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938699    2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938705    2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938709    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938712    2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:37:59.938715    2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.938813    2567 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.941358    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.942564    2567 server.go:1014] "Starting client certificate rotation"
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.942666    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.942709    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.945205    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:37:59.948840 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.947041    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:37:59.960592 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.960579    2567 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:37:59.965014 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.964995    2567 log.go:25] "Validated CRI v1 image API"
Oct 30 00:37:59.965859 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.965829    2567 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:37:59.970937 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.970915    2567 fs.go:135] Filesystem UUIDs: map[2b6d4360-30aa-4302-98ac-683b42a7b635:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d83562f5-ef61-4b9d-97b5-257dcd3a6603:/dev/nvme0n1p4]
Oct 30 00:37:59.970996 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.970938    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 00:37:59.977337 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.977202    2567 manager.go:217] Machine: {Timestamp:2026-10-30 00:37:59.975728697 +0000 UTC m=+0.315856024 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3276383 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d738bed7b6c55377b4e3df6637422 SystemUUID:ec2d738b-ed7b-6c55-377b-4e3df6637422 BootID:07189e76-8efe-4e7a-8f38-4e70b1aa2a7b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:74:1d:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:74:1d:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:b6:5a:7a:0e:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:37:59.977989 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.977977    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:37:59.978085 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.978074    2567 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:37:59.980965 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.980934    2567 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:37:59.981104 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.980969    2567 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:37:59.981145 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.981110    2567 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:37:59.981145 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.981117    2567 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:37:59.981193 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.981152    2567 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:37:59.981193 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.981170    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:37:59.982437 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.982427    2567 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:37:59.987243 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.987222    2567 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:37:59.987369 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.987359    2567 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:37:59.988182 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.988173    2567 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:37:59.988210 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.988190    2567 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:37:59.988210 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.988200    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:37:59.988210 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.988209    2567 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:37:59.988342 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.988217    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:37:59.990641 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.990614    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:37:59.993534 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.993515    2567 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:37:59.998906 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.998893    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:37:59.999865 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.999853    2567 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:37:59.999899 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:37:59.999884    2567 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:38:00.000717 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.000706    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:38:00.000754 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.000724    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:38:00.000754 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.000730    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:38:00.000754 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.000736    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:38:00.000754 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.000743    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:38:00.000754 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.000748    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:38:00.001453 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.001444    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:38:00.001522 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.001454    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:38:00.001522 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.001462    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:38:00.001522 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.001467    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:38:00.001522 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.001476    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:38:00.001522 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.001485    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:38:00.002165 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.002156    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:38:00.002197 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.002165    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:38:00.006134 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.006118    2567 server.go:1263] "Started kubelet"
Oct 30 00:38:00.006251 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.006203    2567 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:38:00.006345 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.006279    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:38:00.006410 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.006363    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:38:00.006856 ip-10-0-130-5 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:38:00.008021 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.007951    2567 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:38:00.008455 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.008440    2567 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:38:00.012665 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.012647    2567 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 00:38:00.015607 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.015588    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:38:00.016248 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.016214    2567 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:38:00.016355 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.016314    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.017119 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017093    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:38:00.017209 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017137    2567 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:38:00.017209 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017150    2567 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:38:00.017330 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017211    2567 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:38:00.017330 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017225    2567 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:38:00.017583 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017568    2567 factory.go:55] Registering systemd factory
Oct 30 00:38:00.017655 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017586    2567 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:38:00.017851 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017834    2567 factory.go:153] Registering CRI-O factory
Oct 30 00:38:00.017851 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017853    2567 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:38:00.018000 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017899    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:38:00.018000 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017912    2567 factory.go:103] Registering Raw factory
Oct 30 00:38:00.018000 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.017942    2567 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:38:00.018932 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.018915    2567 manager.go:319] Starting recovery of all containers
Oct 30 00:38:00.029906 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.029874    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:38:00.030525 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.030507    2567 manager.go:324] Recovery completed
Oct 30 00:38:00.034471 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.034452    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:38:00.036246 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.036213    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:38:00.036327 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.036309    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:38:00.036327 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.036322    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:38:00.036850 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.036837    2567 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:38:00.036850 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.036848    2567 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:38:00.036942 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.036838    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:38:00.036942 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.036864    2567 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:38:00.037539 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.037523    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:38:00.037596 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.037547    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:38:00.037596 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.037561    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:38:00.039049 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.039036    2567 policy_none.go:50] "Start"
Oct 30 00:38:00.039106 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.039052    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:38:00.039106 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.039062    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:38:00.040057 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.040039    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 00:38:00.040451 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.040432    2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:38:00.040540 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.040488    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d5k4k"
Oct 30 00:38:00.041188 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.040048    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-5.ec2.internal.18e327a8341fd07e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-5.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-5.ec2.internal,},FirstTimestamp:2026-10-30 00:38:00.00607859 +0000 UTC m=+0.346205899,LastTimestamp:2026-10-30 00:38:00.00607859 +0000 UTC m=+0.346205899,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-5.ec2.internal,}"
Oct 30 00:38:00.041798 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.041783    2567 policy_none.go:44] "Start"
Oct 30 00:38:00.052246 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.052213    2567 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 00:38:00.055563 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.055492    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-5.ec2.internal.18e327a835ecdb0d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-5.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-5.ec2.internal,},FirstTimestamp:2026-10-30 00:38:00.036293389 +0000 UTC m=+0.376420701,LastTimestamp:2026-10-30 00:38:00.036293389 +0000 UTC m=+0.376420701,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-5.ec2.internal,}"
Oct 30 00:38:00.056207 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.056187    2567 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 00:38:00.056293 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.056186    2567 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 00:38:00.056293 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.056186    2567 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 00:38:00.076706 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.076527    2567 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:38:00.092901 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.092884    2567 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:38:00.093021 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.092923    2567 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:38:00.093021 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.092932    2567 server.go:87] "Starting device plugin registration server"
Oct 30 00:38:00.093205 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.093193    2567 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:38:00.093267 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.093208    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:38:00.093351 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.093327    2567 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:38:00.093482 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.093411    2567 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:38:00.093482 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.093419    2567 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:38:00.093874 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.093852    2567 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:38:00.093948 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.093918    2567 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.166364 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.166334    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:38:00.166364 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.166364    2567 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:38:00.168467 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.166387    2567 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:38:00.168467 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.166433    2567 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 00:38:00.181309 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.181252    2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:00.193485 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.193461    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:38:00.194359 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.194344    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:38:00.194441 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.194370    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:38:00.194441 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.194379    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:38:00.194441 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.194407    2567 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.213507 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.213482    2567 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.213572 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.213508    2567 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-5.ec2.internal\": node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.255563 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.255540    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.266562 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.266543    2567 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"]
Oct 30 00:38:00.266628 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.266618    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:38:00.267502 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.267485    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:38:00.267577 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.267518    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:38:00.267577 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.267530    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:38:00.269808 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.269795    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:38:00.269948 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.269934    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.270002 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.269963    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:38:00.270494 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.270473    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:38:00.270584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.270509    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:38:00.270584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.270520    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:38:00.270584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.270530    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:38:00.270584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.270555    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:38:00.270584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.270568    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:38:00.272633 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.272616    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.272706 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.272649    2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:38:00.273380 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.273366    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:38:00.273453 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.273393    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:38:00.273453 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.273402    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:38:00.305870 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.305846    2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-5.ec2.internal\" not found" node="ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.308274 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.308258    2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-5.ec2.internal\" not found" node="ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.319534 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.319515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.319625 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.319542    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.319625 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.319558    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3b3e87c8faf211497319716dcdc8a4c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-5.ec2.internal\" (UID: \"f3b3e87c8faf211497319716dcdc8a4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.356081 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.356049    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.420173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.420146    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.420173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.420176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.420400 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.420193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3b3e87c8faf211497319716dcdc8a4c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-5.ec2.internal\" (UID: \"f3b3e87c8faf211497319716dcdc8a4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.420400 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.420220    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3b3e87c8faf211497319716dcdc8a4c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-5.ec2.internal\" (UID: \"f3b3e87c8faf211497319716dcdc8a4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.420400 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.420265    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.420400 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.420278    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.456256 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.456172    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.557044 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.557005    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.609218 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.609179    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.611289 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.611263    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:00.657379 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.657346    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.757883 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.757825    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.858430 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.858397    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:00.942052 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.942021    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 00:38:00.942517 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:00.942165    2567 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:38:00.959299 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:00.959264    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:01.045804 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.045726    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:33:00 +0000 UTC" deadline="2028-06-25 02:33:36.31007231 +0000 UTC"
Oct 30 00:38:01.045804 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.045755    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14497h55m35.264320232s"
Oct 30 00:38:01.060291 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:01.060268    2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:01.123863 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.123835    2567 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:01.217404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.217362    2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:01.237327 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:01.237294    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c5cafbcc82820178483fe6020ab6f7.slice/crio-240f181228572cf3999d7ee380753ddfb4bc862a5d5ad6b5d7471023e9e9ca33 WatchSource:0}: Error finding container 240f181228572cf3999d7ee380753ddfb4bc862a5d5ad6b5d7471023e9e9ca33: Status 404 returned error can't find the container with id 240f181228572cf3999d7ee380753ddfb4bc862a5d5ad6b5d7471023e9e9ca33
Oct 30 00:38:01.237616 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:01.237595    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b3e87c8faf211497319716dcdc8a4c.slice/crio-5a92bd22db7888c525b0fc350f89d7199dc33679ae842a52b2399800919c7dd7 WatchSource:0}: Error finding container 5a92bd22db7888c525b0fc350f89d7199dc33679ae842a52b2399800919c7dd7: Status 404 returned error can't find the container with id 5a92bd22db7888c525b0fc350f89d7199dc33679ae842a52b2399800919c7dd7
Oct 30 00:38:01.239545 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.239519    2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:38:01.242483 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.242464    2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:01.243440 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.243423    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:38:01.262431 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.262415    2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:38:01.483391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.483320    2567 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:01.667798 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.667770    2567 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:01.807599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.807521    2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:01.990426 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:01.990387    2567 apiserver.go:52] "Watching apiserver"
Oct 30 00:38:02.017307 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.017278    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 00:38:02.018815 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.018776    2567 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:38:02.019211 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.019184    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-l4gr4","openshift-network-operator/iptables-alerter-6hsqt","openshift-ovn-kubernetes/ovnkube-node-5klc9","kube-system/global-pull-secret-syncer-4h2rk","kube-system/konnectivity-agent-62frw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d","openshift-cluster-node-tuning-operator/tuned-6sjgl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal","openshift-multus/multus-q5pp9","openshift-multus/network-metrics-daemon-qp4wt","openshift-network-diagnostics/network-check-target-tds5t","kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal","openshift-dns/node-resolver-6j4cc","openshift-image-registry/node-ca-frssv"]
Oct 30 00:38:02.021874 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.021849    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.024418 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.024395    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.026813 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.026793    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.029564 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.029542    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.029654 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.029615    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:02.031859 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.031840    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:02.034163 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.034138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.037131 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.037105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.039816 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.039798    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kg5gj\""
Oct 30 00:38:02.039914 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.039843    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:38:02.039998 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.039954    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:38:02.040057 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.040001    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q2s6t\""
Oct 30 00:38:02.040057 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.039849    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:38:02.040162 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.040057    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:38:02.041158 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.040757    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.041880 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.041396    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:38:02.041880 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.041554    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:38:02.041880 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.041562    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ckcc7\""
Oct 30 00:38:02.041880 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.041560    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:38:02.041880 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.041669    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:38:02.041880 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.041775    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:38:02.041880 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.041843    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:38:02.043468 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.043446    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.045957 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.045936    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:02.046044 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.046009    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:02.048784 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.048636    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:02.048784 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.048702    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:02.051179 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.050808    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.063976 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.063922    2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:38:02.063976 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.063974    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cvlwr"
Oct 30 00:38:02.065088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.065069    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cvlwr"
Oct 30 00:38:02.118393 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.118370    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:38:02.128913 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.128880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129071 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.128921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-registration-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.129071 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.128950    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-etc-kubernetes\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.129071 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129017    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-systemd\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.129238 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6sm\" (UniqueName: \"kubernetes.io/projected/f1d925dd-7561-4ee2-8e42-f38ee71296f6-kube-api-access-7l6sm\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.129238 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129129    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-kubelet-config\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.129238 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfgn\" (UniqueName: \"kubernetes.io/projected/38381a16-212b-4d9c-98c2-6d81e4af3b74-kube-api-access-vlfgn\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.129238 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129218    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysconfig\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.129481 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129245    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-conf\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.129481 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129318    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-systemd-units\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129481 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-var-lib-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129481 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-config\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129481 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129371    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-script-lib\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129481 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129391    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-socket-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.129481 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129407    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfab0af-8070-4ffb-9c66-9c94399393d7-hosts-file\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.129481 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129436    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-netns\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129498    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-ovn\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-node-log\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129535    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbvph\" (UniqueName: \"kubernetes.io/projected/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-kube-api-access-rbvph\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129558    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129577    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-cnibin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-socket-dir-parent\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-run\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129664    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-kubelet\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-systemd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129711    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-system-cni-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129796    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fcdt\" (UniqueName: \"kubernetes.io/projected/3bf88444-b628-4aba-a61e-82ae03285b9e-kube-api-access-5fcdt\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-kubernetes\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.129876 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-lib-modules\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-device-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129916    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-sys-fs\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgzz\" (UniqueName: \"kubernetes.io/projected/9b899a78-408a-433c-8e45-c7dd7499a1f2-kube-api-access-ttgzz\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.129958    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-cni-binary-copy\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130033    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-slash\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130063    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-bin\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-conf-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130150    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38381a16-212b-4d9c-98c2-6d81e4af3b74-host\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-modprobe-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130188    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-tmp\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130210    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-netns\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130234    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-env-overrides\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.130661 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovn-node-metrics-cert\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130321    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130344    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-bin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130366    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-sys\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-multus-daemon-config\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130433    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hlvr\" (UniqueName: \"kubernetes.io/projected/87032b63-645e-4392-bd02-80b1289399c3-kube-api-access-2hlvr\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130683    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d925dd-7561-4ee2-8e42-f38ee71296f6-host-slash\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130710    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99fx\" (UniqueName: \"kubernetes.io/projected/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-kube-api-access-j99fx\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-etc-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130770    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1d925dd-7561-4ee2-8e42-f38ee71296f6-iptables-alerter-script\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130798    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-cnibin\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130823    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69lpl\" (UniqueName: \"kubernetes.io/projected/c7f726d8-6d17-4370-a98c-41a979b549ae-kube-api-access-69lpl\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-host\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130891    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38381a16-212b-4d9c-98c2-6d81e4af3b74-serviceca\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.131399 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130917    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-netd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.130947    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d10c0ce2-a04c-4f4f-9977-db48be2e5504-konnectivity-ca\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131044    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-system-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131070    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-kubelet\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfab0af-8070-4ffb-9c66-9c94399393d7-tmp-dir\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-k8s-cni-cncf-io\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-hostroot\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131245    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131313    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-var-lib-kubelet\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131355    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-tuned\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131448    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zv44\" (UniqueName: \"kubernetes.io/projected/edfab0af-8070-4ffb-9c66-9c94399393d7-kube-api-access-8zv44\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131556    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-log-socket\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-dbus\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d10c0ce2-a04c-4f4f-9977-db48be2e5504-agent-certs\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131710    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-multus\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.132201 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-multus-certs\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.133056 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.131791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.146562 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.146534    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:38:02.147091 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.146786    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:38:02.147404 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147381    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k8zgn\""
Oct 30 00:38:02.147538 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147515    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:38:02.147648 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147537    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:38:02.147648 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147549    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-545pj\""
Oct 30 00:38:02.147648 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147579    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:38:02.147648 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147589    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:38:02.147648 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147628    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:38:02.147648 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147644    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:38:02.147648 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147634    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147642    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tmln4\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147677    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147717    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5s9q7\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147738    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147747    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147751    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2bp5j\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147779    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147785    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147779    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147821    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147827    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:38:02.148061 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.147939    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t5wsq\""
Oct 30 00:38:02.172137 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.172076    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal" event={"ID":"f3b3e87c8faf211497319716dcdc8a4c","Type":"ContainerStarted","Data":"5a92bd22db7888c525b0fc350f89d7199dc33679ae842a52b2399800919c7dd7"}
Oct 30 00:38:02.174046 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.174016    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal" event={"ID":"a2c5cafbcc82820178483fe6020ab6f7","Type":"ContainerStarted","Data":"240f181228572cf3999d7ee380753ddfb4bc862a5d5ad6b5d7471023e9e9ca33"}
Oct 30 00:38:02.232763 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.232725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-netd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.232763 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.232768    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d10c0ce2-a04c-4f4f-9977-db48be2e5504-konnectivity-ca\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:02.233010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.232786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.233010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.232814    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-system-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.232902    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-system-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.232911    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-netd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.233010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.232921    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.233010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.232972    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-kubelet\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233010 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfab0af-8070-4ffb-9c66-9c94399393d7-tmp-dir\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-k8s-cni-cncf-io\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233034    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-kubelet\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233061    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-hostroot\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233093    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-k8s-cni-cncf-io\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233092    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233143    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-var-lib-kubelet\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233160    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-hostroot\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-tuned\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233200    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zv44\" (UniqueName: \"kubernetes.io/projected/edfab0af-8070-4ffb-9c66-9c94399393d7-kube-api-access-8zv44\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-log-socket\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233266    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-dbus\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d10c0ce2-a04c-4f4f-9977-db48be2e5504-agent-certs\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-multus\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233368    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfab0af-8070-4ffb-9c66-9c94399393d7-tmp-dir\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.233385 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-multus-certs\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233410    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233438    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233441    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-log-socket\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-registration-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233488    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-var-lib-kubelet\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-etc-kubernetes\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-systemd\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233547    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6sm\" (UniqueName: \"kubernetes.io/projected/f1d925dd-7561-4ee2-8e42-f38ee71296f6-kube-api-access-7l6sm\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-kubelet-config\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233619    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d10c0ce2-a04c-4f4f-9977-db48be2e5504-konnectivity-ca\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233626    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfgn\" (UniqueName: \"kubernetes.io/projected/38381a16-212b-4d9c-98c2-6d81e4af3b74-kube-api-access-vlfgn\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233716    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysconfig\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233725    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-conf\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233744    2567 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233910    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234088 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233986    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysconfig\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.233975    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-dbus\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234077    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-conf\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234077    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-registration-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234105    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-systemd-units\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234120    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-multus-certs\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234135    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-var-lib-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234153    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-multus\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-config\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234188    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-script-lib\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-socket-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfab0af-8070-4ffb-9c66-9c94399393d7-hosts-file\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234266    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-netns\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-ovn\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-node-log\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234361    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvph\" (UniqueName: \"kubernetes.io/projected/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-kube-api-access-rbvph\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234392    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-cnibin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.234839 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234408    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-socket-dir-parent\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-run\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-kubelet\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-systemd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234525    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234543    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-system-cni-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234560    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234581    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcdt\" (UniqueName: \"kubernetes.io/projected/3bf88444-b628-4aba-a61e-82ae03285b9e-kube-api-access-5fcdt\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234601    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-kubernetes\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-lib-modules\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234646    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-device-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234663    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-sys-fs\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234663    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-ovn\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgzz\" (UniqueName: \"kubernetes.io/projected/9b899a78-408a-433c-8e45-c7dd7499a1f2-kube-api-access-ttgzz\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234701    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-etc-kubernetes\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.235853 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234705    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-cni-binary-copy\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234734    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-slash\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234757    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-bin\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234783    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-conf-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234805    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234827    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38381a16-212b-4d9c-98c2-6d81e4af3b74-host\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-modprobe-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234904    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-tmp\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234926    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-netns\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234952    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.234977    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-env-overrides\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovn-node-metrics-cert\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235030    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235141    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-cni-binary-copy\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235113    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-bin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235200    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-sys\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235251    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-node-log\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.236693 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235254    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-slash\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235280    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-bin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235302    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-systemd\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235350    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-cnibin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235350    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-bin\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235384    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-conf-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235411    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-socket-dir-parent\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235521    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235582    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38381a16-212b-4d9c-98c2-6d81e4af3b74-host\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235583    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-netns\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235583    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfab0af-8070-4ffb-9c66-9c94399393d7-hosts-file\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235667    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-modprobe-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235685    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-socket-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235731    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-systemd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235767    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-netns\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235791    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-sys\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235820    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.235829    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.237584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236036    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-kubelet-config\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236044    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236124    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-multus-daemon-config\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236160    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlvr\" (UniqueName: \"kubernetes.io/projected/87032b63-645e-4392-bd02-80b1289399c3-kube-api-access-2hlvr\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236197    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236257    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-kubelet\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.236287    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236280    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-env-overrides\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236329    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-device-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.236379    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:02.736356947 +0000 UTC m=+3.076484266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.236395    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.236447    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret podName:56a648c9-5e85-49b8-91f3-967fad2836f8 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:02.736430446 +0000 UTC m=+3.076557743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret") pod "global-pull-secret-syncer-4h2rk" (UID: "56a648c9-5e85-49b8-91f3-967fad2836f8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236482    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-system-cni-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236595    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-run\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236871    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-script-lib\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.238391 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236916    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236928    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-sys-fs\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236959    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.236995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d925dd-7561-4ee2-8e42-f38ee71296f6-host-slash\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99fx\" (UniqueName: \"kubernetes.io/projected/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-kube-api-access-j99fx\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237049    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-etc-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237076    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1d925dd-7561-4ee2-8e42-f38ee71296f6-iptables-alerter-script\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237084    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-multus-daemon-config\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237102    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-cnibin\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237154    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d925dd-7561-4ee2-8e42-f38ee71296f6-host-slash\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237202    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-etc-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237244    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lpl\" (UniqueName: \"kubernetes.io/projected/c7f726d8-6d17-4370-a98c-41a979b549ae-kube-api-access-69lpl\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-host\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38381a16-212b-4d9c-98c2-6d81e4af3b74-serviceca\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237689    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1d925dd-7561-4ee2-8e42-f38ee71296f6-iptables-alerter-script\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237748    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-var-lib-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237797    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-kubernetes\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.239173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237820    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-config\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.240082 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237819    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38381a16-212b-4d9c-98c2-6d81e4af3b74-serviceca\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.240082 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237855    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-systemd-units\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.240082 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237899    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-lib-modules\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.240082 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.237954    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-host\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.240082 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.238048    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-cnibin\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.240082 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.238398    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d10c0ce2-a04c-4f4f-9977-db48be2e5504-agent-certs\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:02.240082 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.238643    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-tuned\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.240873 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.240851    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-tmp\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.242445 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.242419    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovn-node-metrics-cert\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.252979 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.252959    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:38:02.253108 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.252988    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:38:02.253108 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.253021    2567 projected.go:196] Error preparing data for projected volume kube-api-access-d6zxt for pod openshift-network-diagnostics/network-check-target-tds5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:02.253108 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.253094    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt podName:40aff960-4844-46db-aa92-2efb92303076 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:02.75307533 +0000 UTC m=+3.093202637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d6zxt" (UniqueName: "kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt") pod "network-check-target-tds5t" (UID: "40aff960-4844-46db-aa92-2efb92303076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:02.254230 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.254164    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zv44\" (UniqueName: \"kubernetes.io/projected/edfab0af-8070-4ffb-9c66-9c94399393d7-kube-api-access-8zv44\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.255795 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.255703    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfgn\" (UniqueName: \"kubernetes.io/projected/38381a16-212b-4d9c-98c2-6d81e4af3b74-kube-api-access-vlfgn\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.255795 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.255730    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvph\" (UniqueName: \"kubernetes.io/projected/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-kube-api-access-rbvph\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.259172 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.259143    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6sm\" (UniqueName: \"kubernetes.io/projected/f1d925dd-7561-4ee2-8e42-f38ee71296f6-kube-api-access-7l6sm\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.260930 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.260901    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgzz\" (UniqueName: \"kubernetes.io/projected/9b899a78-408a-433c-8e45-c7dd7499a1f2-kube-api-access-ttgzz\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.261638 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.261613    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcdt\" (UniqueName: \"kubernetes.io/projected/3bf88444-b628-4aba-a61e-82ae03285b9e-kube-api-access-5fcdt\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.262762 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.262742    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lpl\" (UniqueName: \"kubernetes.io/projected/c7f726d8-6d17-4370-a98c-41a979b549ae-kube-api-access-69lpl\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:02.263096 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.263074    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlvr\" (UniqueName: \"kubernetes.io/projected/87032b63-645e-4392-bd02-80b1289399c3-kube-api-access-2hlvr\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.263896 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.263877    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99fx\" (UniqueName: \"kubernetes.io/projected/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-kube-api-access-j99fx\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.336697 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.336620    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:02.347856 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.347830    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:02.348761 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.348732    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cca4f8_e3c4_4796_bd5a_1fe9d4985408.slice/crio-dd5ca4c021a17936c858c826d1091147e30711ef2e9ac4a0924d701952cde58c WatchSource:0}: Error finding container dd5ca4c021a17936c858c826d1091147e30711ef2e9ac4a0924d701952cde58c: Status 404 returned error can't find the container with id dd5ca4c021a17936c858c826d1091147e30711ef2e9ac4a0924d701952cde58c
Oct 30 00:38:02.357665 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.357634    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfab0af_8070_4ffb_9c66_9c94399393d7.slice/crio-13b536f0aead184c905ca8c26683444e28599553e9300b6892080c9df3f63ae8 WatchSource:0}: Error finding container 13b536f0aead184c905ca8c26683444e28599553e9300b6892080c9df3f63ae8: Status 404 returned error can't find the container with id 13b536f0aead184c905ca8c26683444e28599553e9300b6892080c9df3f63ae8
Oct 30 00:38:02.370991 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.370960    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:02.377373 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.377350    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:02.381627 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.381603    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:02.382896 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.382872    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10c0ce2_a04c_4f4f_9977_db48be2e5504.slice/crio-aab0c2e8f9f7615a708f0505d039b5a06854eb26daa49c0ef0ffd01191a192b0 WatchSource:0}: Error finding container aab0c2e8f9f7615a708f0505d039b5a06854eb26daa49c0ef0ffd01191a192b0: Status 404 returned error can't find the container with id aab0c2e8f9f7615a708f0505d039b5a06854eb26daa49c0ef0ffd01191a192b0
Oct 30 00:38:02.384365 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.384341    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:02.389944 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.389907    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7743261_4eb5_44c4_a6a0_ef3fea5e492c.slice/crio-775233877f80c24f544008201c7230151e8f583c1c7df2598c9865dbd3245ec6 WatchSource:0}: Error finding container 775233877f80c24f544008201c7230151e8f583c1c7df2598c9865dbd3245ec6: Status 404 returned error can't find the container with id 775233877f80c24f544008201c7230151e8f583c1c7df2598c9865dbd3245ec6
Oct 30 00:38:02.393040 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.393018    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b899a78_408a_433c_8e45_c7dd7499a1f2.slice/crio-b42d909c9e24bea099fed03d603ebccf4a27aa47b91d5a5161f301b7fa5d8225 WatchSource:0}: Error finding container b42d909c9e24bea099fed03d603ebccf4a27aa47b91d5a5161f301b7fa5d8225: Status 404 returned error can't find the container with id b42d909c9e24bea099fed03d603ebccf4a27aa47b91d5a5161f301b7fa5d8225
Oct 30 00:38:02.397509 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.397476    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87032b63_645e_4392_bd02_80b1289399c3.slice/crio-e1fd7b895496783c7254d47d97bb48f11a7f1e4ea4a83241f9ac89baeb2965b0 WatchSource:0}: Error finding container e1fd7b895496783c7254d47d97bb48f11a7f1e4ea4a83241f9ac89baeb2965b0: Status 404 returned error can't find the container with id e1fd7b895496783c7254d47d97bb48f11a7f1e4ea4a83241f9ac89baeb2965b0
Oct 30 00:38:02.413186 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.411379    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:02.413571 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.413542    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:02.416927 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.416905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:02.424442 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.424421    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d925dd_7561_4ee2_8e42_f38ee71296f6.slice/crio-f2720d61591632d53e72a27378c6c0a955158c1c54bbd6aaed6f4deda9b5feaf WatchSource:0}: Error finding container f2720d61591632d53e72a27378c6c0a955158c1c54bbd6aaed6f4deda9b5feaf: Status 404 returned error can't find the container with id f2720d61591632d53e72a27378c6c0a955158c1c54bbd6aaed6f4deda9b5feaf
Oct 30 00:38:02.428811 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.428790    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38381a16_212b_4d9c_98c2_6d81e4af3b74.slice/crio-c3fa82872f31f5f151ea52ec1d18a3e46dd820c4dd7012960311453bdba8eecf WatchSource:0}: Error finding container c3fa82872f31f5f151ea52ec1d18a3e46dd820c4dd7012960311453bdba8eecf: Status 404 returned error can't find the container with id c3fa82872f31f5f151ea52ec1d18a3e46dd820c4dd7012960311453bdba8eecf
Oct 30 00:38:02.431510 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:02.431484    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf88444_b628_4aba_a61e_82ae03285b9e.slice/crio-1455a5657ba343086772ff987a745481368d49e09c28041761155497e7398c03 WatchSource:0}: Error finding container 1455a5657ba343086772ff987a745481368d49e09c28041761155497e7398c03: Status 404 returned error can't find the container with id 1455a5657ba343086772ff987a745481368d49e09c28041761155497e7398c03
Oct 30 00:38:02.741173 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.741134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:02.741339 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.741226    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:02.741339 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.741322    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:02.741463 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.741396    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret podName:56a648c9-5e85-49b8-91f3-967fad2836f8 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:03.741378271 +0000 UTC m=+4.081505586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret") pod "global-pull-secret-syncer-4h2rk" (UID: "56a648c9-5e85-49b8-91f3-967fad2836f8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:02.741463 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.741396    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:02.741582 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.741483    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:03.741465477 +0000 UTC m=+4.081592775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:02.842182 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:02.842149    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:02.842368 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.842349    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:38:02.842435 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.842378    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:38:02.842435 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.842393    2567 projected.go:196] Error preparing data for projected volume kube-api-access-d6zxt for pod openshift-network-diagnostics/network-check-target-tds5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:02.842534 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:02.842454    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt podName:40aff960-4844-46db-aa92-2efb92303076 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:03.842435921 +0000 UTC m=+4.182563219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6zxt" (UniqueName: "kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt") pod "network-check-target-tds5t" (UID: "40aff960-4844-46db-aa92-2efb92303076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:03.070396 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.069999    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:33:02 +0000 UTC" deadline="2028-05-18 23:33:44.365443759 +0000 UTC"
Oct 30 00:38:03.070396 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.070305    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13606h55m41.295145077s"
Oct 30 00:38:03.167636 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.167109    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:03.167636 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.167254    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:03.210416 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.210167    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal" event={"ID":"f3b3e87c8faf211497319716dcdc8a4c","Type":"ContainerStarted","Data":"dcccc59a9cb5af431cdb1eef6e073cdfa73b8ab6bcae91379edb1ad96c807486"}
Oct 30 00:38:03.232822 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.232786    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-frssv" event={"ID":"38381a16-212b-4d9c-98c2-6d81e4af3b74","Type":"ContainerStarted","Data":"c3fa82872f31f5f151ea52ec1d18a3e46dd820c4dd7012960311453bdba8eecf"}
Oct 30 00:38:03.234356 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.234326    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hsqt" event={"ID":"f1d925dd-7561-4ee2-8e42-f38ee71296f6","Type":"ContainerStarted","Data":"f2720d61591632d53e72a27378c6c0a955158c1c54bbd6aaed6f4deda9b5feaf"}
Oct 30 00:38:03.235877 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.235833    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q5pp9" event={"ID":"87032b63-645e-4392-bd02-80b1289399c3","Type":"ContainerStarted","Data":"e1fd7b895496783c7254d47d97bb48f11a7f1e4ea4a83241f9ac89baeb2965b0"}
Oct 30 00:38:03.237535 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.237500    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"775233877f80c24f544008201c7230151e8f583c1c7df2598c9865dbd3245ec6"}
Oct 30 00:38:03.242286 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.241295    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6sjgl" event={"ID":"04cca4f8-e3c4-4796-bd5a-1fe9d4985408","Type":"ContainerStarted","Data":"dd5ca4c021a17936c858c826d1091147e30711ef2e9ac4a0924d701952cde58c"}
Oct 30 00:38:03.243535 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.243149    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerStarted","Data":"1455a5657ba343086772ff987a745481368d49e09c28041761155497e7398c03"}
Oct 30 00:38:03.245829 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.245711    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d" event={"ID":"9b899a78-408a-433c-8e45-c7dd7499a1f2","Type":"ContainerStarted","Data":"b42d909c9e24bea099fed03d603ebccf4a27aa47b91d5a5161f301b7fa5d8225"}
Oct 30 00:38:03.249168 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.249036    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal" podStartSLOduration=2.249020335 podStartE2EDuration="2.249020335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:03.247935618 +0000 UTC m=+3.588062950" watchObservedRunningTime="2026-10-30 00:38:03.249020335 +0000 UTC m=+3.589147655"
Oct 30 00:38:03.253682 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.253596    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-62frw" event={"ID":"d10c0ce2-a04c-4f4f-9977-db48be2e5504","Type":"ContainerStarted","Data":"aab0c2e8f9f7615a708f0505d039b5a06854eb26daa49c0ef0ffd01191a192b0"}
Oct 30 00:38:03.264780 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.264747    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6j4cc" event={"ID":"edfab0af-8070-4ffb-9c66-9c94399393d7","Type":"ContainerStarted","Data":"13b536f0aead184c905ca8c26683444e28599553e9300b6892080c9df3f63ae8"}
Oct 30 00:38:03.753291 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.753191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:03.753291 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.753283    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:03.753511 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.753431    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:03.753511 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.753489    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret podName:56a648c9-5e85-49b8-91f3-967fad2836f8 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:05.753471594 +0000 UTC m=+6.093598894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret") pod "global-pull-secret-syncer-4h2rk" (UID: "56a648c9-5e85-49b8-91f3-967fad2836f8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:03.753968 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.753946    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:03.754057 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.754007    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:05.753990419 +0000 UTC m=+6.094117719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:03.854349 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:03.854311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:03.854523 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.854498    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:38:03.854523 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.854520    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:38:03.854665 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.854533    2567 projected.go:196] Error preparing data for projected volume kube-api-access-d6zxt for pod openshift-network-diagnostics/network-check-target-tds5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:03.854665 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:03.854594    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt podName:40aff960-4844-46db-aa92-2efb92303076 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:05.85457604 +0000 UTC m=+6.194703351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6zxt" (UniqueName: "kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt") pod "network-check-target-tds5t" (UID: "40aff960-4844-46db-aa92-2efb92303076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:04.071379 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:04.071262    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:33:02 +0000 UTC" deadline="2028-07-27 18:39:25.989151426 +0000 UTC"
Oct 30 00:38:04.071379 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:04.071299    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15282h1m21.917856815s"
Oct 30 00:38:04.170997 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:04.170967    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:04.180601 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:04.180507    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:04.180897 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:04.180673    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:04.180897 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:04.180828    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:04.270906 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:04.270860    2567 generic.go:356] "Generic (PLEG): container finished" podID="a2c5cafbcc82820178483fe6020ab6f7" containerID="b8f81471a9b31c7da9b69dd149585eb16c66f99659320a1dbd51d3e9fc1ed7c9" exitCode=0
Oct 30 00:38:04.271899 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:04.271293    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal" event={"ID":"a2c5cafbcc82820178483fe6020ab6f7","Type":"ContainerDied","Data":"b8f81471a9b31c7da9b69dd149585eb16c66f99659320a1dbd51d3e9fc1ed7c9"}
Oct 30 00:38:05.167837 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:05.167327    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:05.167837 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.167463    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:05.280503 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:05.280463    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal" event={"ID":"a2c5cafbcc82820178483fe6020ab6f7","Type":"ContainerStarted","Data":"ac6899c82ad5f0d671cbd9f96dc5a62f8bd85a30299396b862291e6992ffb99c"}
Oct 30 00:38:05.283780 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:05.283748    2567 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="e6d226d9f7bf30fd1a078d1c8f7311b70721a42616b5e46b26c42a9a1bb5fc56" exitCode=0
Oct 30 00:38:05.283919 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:05.283796    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerDied","Data":"e6d226d9f7bf30fd1a078d1c8f7311b70721a42616b5e46b26c42a9a1bb5fc56"}
Oct 30 00:38:05.361218 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:05.360783    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal" podStartSLOduration=4.360764972 podStartE2EDuration="4.360764972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:05.317911998 +0000 UTC m=+5.658039317" watchObservedRunningTime="2026-10-30 00:38:05.360764972 +0000 UTC m=+5.700892291"
Oct 30 00:38:05.770764 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:05.770663    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:05.770764 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:05.770742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:05.770982 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.770893    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:05.770982 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.770953    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret podName:56a648c9-5e85-49b8-91f3-967fad2836f8 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:09.770934937 +0000 UTC m=+10.111062237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret") pod "global-pull-secret-syncer-4h2rk" (UID: "56a648c9-5e85-49b8-91f3-967fad2836f8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:05.771380 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.771360    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:05.771468 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.771417    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:09.771400966 +0000 UTC m=+10.111528279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:05.871282 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:05.871225    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:05.871528 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.871427    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:38:05.871528 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.871446    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:38:05.871528 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.871459    2567 projected.go:196] Error preparing data for projected volume kube-api-access-d6zxt for pod openshift-network-diagnostics/network-check-target-tds5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:05.871528 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:05.871520    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt podName:40aff960-4844-46db-aa92-2efb92303076 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:09.871500724 +0000 UTC m=+10.211628041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6zxt" (UniqueName: "kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt") pod "network-check-target-tds5t" (UID: "40aff960-4844-46db-aa92-2efb92303076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:06.168763 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:06.168691    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:06.169268 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:06.168823    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:06.169268 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:06.169178    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:06.169389 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:06.169302    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:07.167455 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:07.167426    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:07.167611 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:07.167531    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:07.297447 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:07.297414    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-62frw" event={"ID":"d10c0ce2-a04c-4f4f-9977-db48be2e5504","Type":"ContainerStarted","Data":"a9c62686b5d512234258c27293422393e610ba33fee8e82b81675fa1540adda6"}
Oct 30 00:38:07.336799 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:07.336602    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-62frw" podStartSLOduration=3.494202231 podStartE2EDuration="7.336582682s" totalImagesPullingTime="3.842380451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:07.335623858 +0000 UTC m=+7.675751177" watchObservedRunningTime="2026-10-30 00:38:07.336582682 +0000 UTC m=+7.676710002"
Oct 30 00:38:08.167854 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:08.167368    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:08.167854 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:08.167502    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:08.168383 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:08.168162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:08.168383 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:08.168293    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:08.304995 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:08.304919    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-frssv" event={"ID":"38381a16-212b-4d9c-98c2-6d81e4af3b74","Type":"ContainerStarted","Data":"c0efc37755cfbf2c69920f2a06b40cac41a017533e52245a126b7e8055f4d562"}
Oct 30 00:38:08.309758 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:08.309724    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d" event={"ID":"9b899a78-408a-433c-8e45-c7dd7499a1f2","Type":"ContainerStarted","Data":"8875c20ac173deb7359c876e01d1180b656e2c6541ba4ae65bf652a7cb3b1c92"}
Oct 30 00:38:09.167267 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:09.167219    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:09.167449 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.167366    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:09.476078 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:09.476049    2567 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:38:09.804669 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:09.804591    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:09.804669 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:09.804662    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:09.804876 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.804804    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:09.804876 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.804864    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:17.804847096 +0000 UTC m=+18.144974406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:09.820453 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.820417    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:09.820615 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.820523    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret podName:56a648c9-5e85-49b8-91f3-967fad2836f8 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:17.820501628 +0000 UTC m=+18.160628942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret") pod "global-pull-secret-syncer-4h2rk" (UID: "56a648c9-5e85-49b8-91f3-967fad2836f8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:09.905487 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:09.905000    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:09.905487 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.905191    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:38:09.905487 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.905209    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:38:09.905487 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.905221    2567 projected.go:196] Error preparing data for projected volume kube-api-access-d6zxt for pod openshift-network-diagnostics/network-check-target-tds5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:09.905487 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:09.905293    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt podName:40aff960-4844-46db-aa92-2efb92303076 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:17.905274872 +0000 UTC m=+18.245402194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6zxt" (UniqueName: "kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt") pod "network-check-target-tds5t" (UID: "40aff960-4844-46db-aa92-2efb92303076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:10.104833 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.103606    2567 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:38:09.476073965Z","UUID":"fa3b5739-efcd-4cc5-a1b3-f7902eefea40","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:38:10.106309 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.106287    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:38:10.106402 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.106320    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:38:10.170396 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.170356    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:10.170555 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:10.170499    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:10.171134 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.171112    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:10.171300 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:10.171275    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:10.317194 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.317098    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q5pp9" event={"ID":"87032b63-645e-4392-bd02-80b1289399c3","Type":"ContainerStarted","Data":"dc01bb9cfccb2521748c8adf6ad7e6fb1fa9b9f6ca1969a3fbb4eddeb7ebff30"}
Oct 30 00:38:10.321833 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.321312    2567 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="38ac1e2c28fac97eb69912552583f2b54cfec45f15bc071affd21141eaafd810" exitCode=0
Oct 30 00:38:10.321833 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.321408    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerDied","Data":"38ac1e2c28fac97eb69912552583f2b54cfec45f15bc071affd21141eaafd810"}
Oct 30 00:38:10.328081 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.328033    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d" event={"ID":"9b899a78-408a-433c-8e45-c7dd7499a1f2","Type":"ContainerStarted","Data":"2bc7b93b6a32686e5edc1364a4cb498a8e6221a0215cc10c28a1670703a6ba5a"}
Oct 30 00:38:10.356611 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.356147    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-q5pp9" podStartSLOduration=2.862740002 podStartE2EDuration="10.356131103s" totalImagesPullingTime="7.493391101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:10.355763019 +0000 UTC m=+10.695890338" watchObservedRunningTime="2026-10-30 00:38:10.356131103 +0000 UTC m=+10.696258427"
Oct 30 00:38:10.356611 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:10.356398    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-frssv" podStartSLOduration=5.482126962 podStartE2EDuration="10.356391614s" totalImagesPullingTime="4.874264652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:08.348323316 +0000 UTC m=+8.688450634" watchObservedRunningTime="2026-10-30 00:38:10.356391614 +0000 UTC m=+10.696518932"
Oct 30 00:38:11.167457 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.167058    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:11.167796 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:11.167497    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:11.333609 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.333521    2567 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="3440ce8f72042d72436921cd5bd7779d645e02bf5d27785a553e198f3fe992b4" exitCode=0
Oct 30 00:38:11.333767 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.333615    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerDied","Data":"3440ce8f72042d72436921cd5bd7779d645e02bf5d27785a553e198f3fe992b4"}
Oct 30 00:38:11.337188 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.337139    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d" event={"ID":"9b899a78-408a-433c-8e45-c7dd7499a1f2","Type":"ContainerStarted","Data":"5d555301c7d271589d4f13f65d2ee91c7af5081cb4e9b6debe28a972a5ff0f2c"}
Oct 30 00:38:11.339373 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.339348    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6j4cc" event={"ID":"edfab0af-8070-4ffb-9c66-9c94399393d7","Type":"ContainerStarted","Data":"d3bb94f75c5e6eecfccc8ab69406327fe4f7beb7b5b69c2b28975280731f249b"}
Oct 30 00:38:11.412897 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.412847    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6j4cc" podStartSLOduration=3.177824935 podStartE2EDuration="11.41283344s" totalImagesPullingTime="8.235008505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:11.412698207 +0000 UTC m=+11.752825525" watchObservedRunningTime="2026-10-30 00:38:11.41283344 +0000 UTC m=+11.752960751"
Oct 30 00:38:11.451226 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.451179    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d" podStartSLOduration=3.7039714740000003 podStartE2EDuration="11.451160078s" totalImagesPullingTime="7.747188604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:11.450929721 +0000 UTC m=+11.791057040" watchObservedRunningTime="2026-10-30 00:38:11.451160078 +0000 UTC m=+11.791287398"
Oct 30 00:38:11.482402 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.482372    2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:11.482894 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:11.482875    2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:12.168917 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:12.168839    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:12.168917 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:12.168871    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:12.169441 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:12.168991    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:12.169441 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:12.169082    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:12.343613 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:12.343578    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hsqt" event={"ID":"f1d925dd-7561-4ee2-8e42-f38ee71296f6","Type":"ContainerStarted","Data":"9f8b3b901d2bca59a515ff25bf589bfc45639dcbcabb2b76d9c77afe4489f7d0"}
Oct 30 00:38:12.346369 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:12.346340    2567 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="078ff424f5be733e825c12a1b080e4f9c1a1b98cdbb5fdb973057591b386f01d" exitCode=0
Oct 30 00:38:12.346511 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:12.346443    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerDied","Data":"078ff424f5be733e825c12a1b080e4f9c1a1b98cdbb5fdb973057591b386f01d"}
Oct 30 00:38:12.346743 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:12.346725    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:12.347584 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:12.347077    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:12.384282 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:12.384225    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6hsqt" podStartSLOduration=4.37139073 podStartE2EDuration="12.384211254s" totalImagesPullingTime="8.012820524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:12.384184069 +0000 UTC m=+12.724311387" watchObservedRunningTime="2026-10-30 00:38:12.384211254 +0000 UTC m=+12.724338572"
Oct 30 00:38:13.167403 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:13.166884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:13.167403 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:13.166996    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:13.350987 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:13.350947    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6sjgl" event={"ID":"04cca4f8-e3c4-4796-bd5a-1fe9d4985408","Type":"ContainerStarted","Data":"edb92d7ff87075fda35a9ea8a3a818887e60fa1191a9251d22c116e2ec3d9e8d"}
Oct 30 00:38:13.404940 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:13.404884    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6sjgl" podStartSLOduration=3.042969938 podStartE2EDuration="13.404870805s" totalImagesPullingTime="10.361900867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:13.404633807 +0000 UTC m=+13.744761125" watchObservedRunningTime="2026-10-30 00:38:13.404870805 +0000 UTC m=+13.744998152"
Oct 30 00:38:14.167185 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:14.167158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:14.167395 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:14.167344    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:14.167395 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:14.167359    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:14.167528 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:14.167476    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:14.356710 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:14.356674    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerStarted","Data":"b0ed4bb00df473b7e086284ef6f7aeeead78bbe6f918b5a2ac18d16c974653a0"}
Oct 30 00:38:15.166646 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:15.166605    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:15.166815 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:15.166749    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:15.361540 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:15.361509    2567 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="b0ed4bb00df473b7e086284ef6f7aeeead78bbe6f918b5a2ac18d16c974653a0" exitCode=0
Oct 30 00:38:15.362133 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:15.361573    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerDied","Data":"b0ed4bb00df473b7e086284ef6f7aeeead78bbe6f918b5a2ac18d16c974653a0"}
Oct 30 00:38:16.167626 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:16.167404    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:16.167817 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:16.167499    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:16.167817 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:16.167753    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:16.167955 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:16.167836    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:16.366913 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:16.366883    2567 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="26fe6c3a0fec07d013f33134490870d0965768e58d511d396ab8b2b715fe2cc4" exitCode=0
Oct 30 00:38:16.367287 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:16.366935    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerDied","Data":"26fe6c3a0fec07d013f33134490870d0965768e58d511d396ab8b2b715fe2cc4"}
Oct 30 00:38:17.166831 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.166662    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:17.167020 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.166906    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:17.371923 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.371845    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"c95ac161b3f26b028797c4d1e696f76d312bf7efdb174ddb75b9a8de3088654f"}
Oct 30 00:38:17.371923 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.371914    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"d17040b4ec0a923d408047a8fb2446b5d7b964283b66ebf6be5d302e611058fe"}
Oct 30 00:38:17.372362 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.371944    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"9ecd6b20fb34be2960d9c8c375088c91dc25f7834f08d3580b85d4573fbbba23"}
Oct 30 00:38:17.372362 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.371980    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"39bb826fe09e0d23d636ba1e4b5df0983bebaa76a804f6965d340c777021277a"}
Oct 30 00:38:17.372362 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.372007    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"d5118e25a7274678aa44484aeaf31486e3edf5eab3ec5418162ba151f6a08eb1"}
Oct 30 00:38:17.372362 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.372040    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"464c92ac605ea8d48a86d5d2aa39240bc2bbe7d2c6fec946c3b72b05ec2315ee"}
Oct 30 00:38:17.375116 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.375089    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" event={"ID":"3bf88444-b628-4aba-a61e-82ae03285b9e","Type":"ContainerStarted","Data":"117e99c10165fbfa54b5f3de3aefaec64cde7f17bd21917f851610b6079f60a7"}
Oct 30 00:38:17.417551 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.417498    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l4gr4" podStartSLOduration=7.302451783 podStartE2EDuration="17.417484541s" totalImagesPullingTime="10.115032758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:17.417141554 +0000 UTC m=+17.757268892" watchObservedRunningTime="2026-10-30 00:38:17.417484541 +0000 UTC m=+17.757611859"
Oct 30 00:38:17.869101 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.869019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:17.869101 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.869082    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:17.869300 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.869195    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:17.869300 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.869210    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:17.869300 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.869276    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret podName:56a648c9-5e85-49b8-91f3-967fad2836f8 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:33.869261057 +0000 UTC m=+34.209388353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret") pod "global-pull-secret-syncer-4h2rk" (UID: "56a648c9-5e85-49b8-91f3-967fad2836f8") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:38:17.869300 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.869291    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:33.869284468 +0000 UTC m=+34.209411763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:38:17.969625 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:17.969588    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:17.969781 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.969746    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:38:17.969781 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.969769    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:38:17.969781 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.969779    2567 projected.go:196] Error preparing data for projected volume kube-api-access-d6zxt for pod openshift-network-diagnostics/network-check-target-tds5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:17.969869 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:17.969826    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt podName:40aff960-4844-46db-aa92-2efb92303076 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:33.969812887 +0000 UTC m=+34.309940185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6zxt" (UniqueName: "kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt") pod "network-check-target-tds5t" (UID: "40aff960-4844-46db-aa92-2efb92303076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:38:18.166936 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:18.166855    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:18.166936 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:18.166879    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:18.167145 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:18.166960    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:18.167145 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:18.167087    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:19.166918 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:19.166889    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:19.167293 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:19.166989    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:19.382947 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:19.382871    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"5d51e100bc0ef30e8b1bb3a31adf43dc4ab44016123c87a777255cad66e36a4a"}
Oct 30 00:38:20.169875 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:20.169849    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:20.170342 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:20.169966    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:20.170342 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:20.170009    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:20.170342 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:20.170091    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:21.167022 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:21.166999    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:21.167115 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:21.167097    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:21.390322 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:21.390245    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" event={"ID":"f7743261-4eb5-44c4-a6a0-ef3fea5e492c","Type":"ContainerStarted","Data":"75b23c6ad4ec244eaae95a2a823598f5645d75d04c34cb2f1062a9369a0edc5c"}
Oct 30 00:38:21.391067 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:21.390466    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:21.391067 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:21.390537    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:21.406434 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:21.406408    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:21.406568 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:21.406529    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:21.471060 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:21.470999    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9" podStartSLOduration=7.180874329 podStartE2EDuration="21.470962827s" totalImagesPullingTime="14.290088498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:21.470927584 +0000 UTC m=+21.811054928" watchObservedRunningTime="2026-10-30 00:38:21.470962827 +0000 UTC m=+21.811090138"
Oct 30 00:38:22.167246 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:22.167211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:22.167428 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:22.167221    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:22.167428 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:22.167343    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:22.167517 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:22.167430    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:22.392920 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:22.392891    2567 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:38:23.167152 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:23.166970    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:23.167337 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:23.167292    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:23.309357 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:23.309227    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tds5t"]
Oct 30 00:38:23.310077 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:23.310055    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4h2rk"]
Oct 30 00:38:23.310177 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:23.310155    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:23.310278 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:23.310258    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:23.320330 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:23.320297    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qp4wt"]
Oct 30 00:38:23.320411 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:23.320393    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:23.320489 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:23.320474    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:23.394690 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:23.394662    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:23.395049 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:23.394734    2567 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:38:23.395049 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:23.394757    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:24.712532 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:24.712495    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:24.896758 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:24.896728    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:25.167154 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:25.167066    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:25.167316 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:25.167081    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:25.167316 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:25.167189    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:25.167316 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:25.167221    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:25.167448 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:25.167311    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:25.167448 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:25.167397    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:27.166643 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:27.166608    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:27.166643 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:27.166633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:27.167096 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:27.166633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:27.167096 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:27.166726    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:27.167096 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:27.166783    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:27.167096 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:27.166884    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:28.548245 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:28.548202    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:29.167274 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.167207    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:29.167274 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.167249    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:29.167547 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.167351    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:29.167547 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:29.167371    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qp4wt" podUID="c7f726d8-6d17-4370-a98c-41a979b549ae"
Oct 30 00:38:29.167547 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:29.167440    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tds5t" podUID="40aff960-4844-46db-aa92-2efb92303076"
Oct 30 00:38:29.167547 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:29.167504    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4h2rk" podUID="56a648c9-5e85-49b8-91f3-967fad2836f8"
Oct 30 00:38:29.807923 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.807890    2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeReady"
Oct 30 00:38:29.808329 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.808000    2567 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 00:38:29.891638 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.891604    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mnj2x"]
Oct 30 00:38:29.895752 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.895735    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:29.896915 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.896893    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k54rc"]
Oct 30 00:38:29.899325 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.899309    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:29.909645 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.909627    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:38:29.909748 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.909660    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jxbz5\""
Oct 30 00:38:29.910331 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.910314    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:38:29.910419 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.910406    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:38:29.914533 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.914514    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mnj2x"]
Oct 30 00:38:29.944403 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:29.944375    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k54rc"]
Oct 30 00:38:30.011163 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.011131    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:38:30.011163 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.011158    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:38:30.011384 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.011277    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7gr5j\""
Oct 30 00:38:30.053724 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.053690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.053882 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.053819    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-cert\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:30.053882 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.053864    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f3661-225a-43e3-8f77-5122261f257f-config-volume\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.053960 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.053890    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82s67\" (UniqueName: \"kubernetes.io/projected/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-kube-api-access-82s67\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:30.053960 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.053913    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49br2\" (UniqueName: \"kubernetes.io/projected/1b9f3661-225a-43e3-8f77-5122261f257f-kube-api-access-49br2\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.054064 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.053998    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9f3661-225a-43e3-8f77-5122261f257f-tmp-dir\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.080919 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.080855    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:30.155110 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.155080    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f3661-225a-43e3-8f77-5122261f257f-config-volume\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.155110 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.155114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82s67\" (UniqueName: \"kubernetes.io/projected/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-kube-api-access-82s67\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:30.155362 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.155133    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49br2\" (UniqueName: \"kubernetes.io/projected/1b9f3661-225a-43e3-8f77-5122261f257f-kube-api-access-49br2\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.155362 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.155158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9f3661-225a-43e3-8f77-5122261f257f-tmp-dir\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.155362 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.155187    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.155362 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.155267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-cert\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:30.155554 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:30.155366    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:38:30.155554 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:30.155434    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls podName:1b9f3661-225a-43e3-8f77-5122261f257f nodeName:}" failed. No retries permitted until 2026-10-30 00:38:30.65541432 +0000 UTC m=+30.995541633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls") pod "dns-default-k54rc" (UID: "1b9f3661-225a-43e3-8f77-5122261f257f") : secret "dns-default-metrics-tls" not found
Oct 30 00:38:30.155656 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.155568    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9f3661-225a-43e3-8f77-5122261f257f-tmp-dir\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.155803 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.155783    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f3661-225a-43e3-8f77-5122261f257f-config-volume\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.159050 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.159032    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-cert\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:30.184602 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.184573    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49br2\" (UniqueName: \"kubernetes.io/projected/1b9f3661-225a-43e3-8f77-5122261f257f-kube-api-access-49br2\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.185738 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.185717    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82s67\" (UniqueName: \"kubernetes.io/projected/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-kube-api-access-82s67\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:30.209605 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.209577    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:30.211424 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.211402    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:30.410753 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.408409    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:30.532089 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.532059    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:30.545847 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.545812    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mnj2x"]
Oct 30 00:38:30.550691 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:30.550665    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bbf453_22c0_42ae_843d_7a9fafe9f7c7.slice/crio-7160b009a1359edb5160872ba3c9db507ea784b3d33c4b51daaae1b2c5a44f31 WatchSource:0}: Error finding container 7160b009a1359edb5160872ba3c9db507ea784b3d33c4b51daaae1b2c5a44f31: Status 404 returned error can't find the container with id 7160b009a1359edb5160872ba3c9db507ea784b3d33c4b51daaae1b2c5a44f31
Oct 30 00:38:30.577782 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.577763    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7tqzf"]
Oct 30 00:38:30.598599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.598580    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.621938 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.621913    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7tqzf"]
Oct 30 00:38:30.623571 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.623545    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:38:30.623671 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.623651    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqzrd\""
Oct 30 00:38:30.658850 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.658824    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:30.658966 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:30.658932    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:38:30.659000 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:30.658980    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls podName:1b9f3661-225a-43e3-8f77-5122261f257f nodeName:}" failed. No retries permitted until 2026-10-30 00:38:31.658966915 +0000 UTC m=+31.999094212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls") pod "dns-default-k54rc" (UID: "1b9f3661-225a-43e3-8f77-5122261f257f") : secret "dns-default-metrics-tls" not found
Oct 30 00:38:30.662353 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.662297    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9wq8z-zs5sd"]
Oct 30 00:38:30.697168 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.697146    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9wq8z-zs5sd"]
Oct 30 00:38:30.697298 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.697255    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.702971 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.702940    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:30.711496 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.711477    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:38:30.711624 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.711614    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:38:30.711696 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.711631    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:38:30.711868 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.711853    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 00:38:30.724336 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.724315    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-98rrr\""
Oct 30 00:38:30.724509 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.724494    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 00:38:30.759883 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.759851    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.759883 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.759879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.760046 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.759918    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tld2q\" (UniqueName: \"kubernetes.io/projected/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-api-access-tld2q\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.760046 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.760031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a312a1ca-eba6-4a80-b070-caee6de84c04-crio-socket\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.760107 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.760100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a312a1ca-eba6-4a80-b070-caee6de84c04-data-volume\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.848700 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.848669    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:30.857125 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.857102    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"]
Oct 30 00:38:30.860571 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.860546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tld2q\" (UniqueName: \"kubernetes.io/projected/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-api-access-tld2q\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.860674 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.860592    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.860724 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.860703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a312a1ca-eba6-4a80-b070-caee6de84c04-crio-socket\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.860881 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.860864    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.860942 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.860879    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a312a1ca-eba6-4a80-b070-caee6de84c04-crio-socket\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.860942 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.860888    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a312a1ca-eba6-4a80-b070-caee6de84c04-data-volume\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.860942 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.860935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.861087 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.860953    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.861087 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:30.860991    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:38:30.861087 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:30.861018    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:30.861087 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:30.861056    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:31.361038213 +0000 UTC m=+31.701165524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : configmap "kube-rbac-proxy" not found
Oct 30 00:38:30.861087 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:30.861076    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:31.361066332 +0000 UTC m=+31.701193633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:30.861309 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.861116    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.861309 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.861130    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a312a1ca-eba6-4a80-b070-caee6de84c04-data-volume\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.861309 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.861142    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552tl\" (UniqueName: \"kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.870998 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.870976    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"
Oct 30 00:38:30.883540 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.883511    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"]
Oct 30 00:38:30.885732 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.885711    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tld2q\" (UniqueName: \"kubernetes.io/projected/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-api-access-tld2q\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:30.892423 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.892405    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:38:30.892520 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.892489    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2jw8t\""
Oct 30 00:38:30.892563 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.892525    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:38:30.961653 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.961623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.961786 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.961695    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.961786 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.961722    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552tl\" (UniqueName: \"kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.961786 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.961756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.961786 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.961784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d24x6\" (UniqueName: \"kubernetes.io/projected/904817ce-d573-4b9e-ba53-646819696439-kube-api-access-d24x6\") pod \"migrator-66df5d496b-sxwqs\" (UID: \"904817ce-d573-4b9e-ba53-646819696439\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"
Oct 30 00:38:30.962164 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.962142    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.962300 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.962285    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.962622 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.962605    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:30.979556 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.979528    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:30.981266 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:30.981249    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552tl\" (UniqueName: \"kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:31.020781 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.020761    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:31.063079 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.063050    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d24x6\" (UniqueName: \"kubernetes.io/projected/904817ce-d573-4b9e-ba53-646819696439-kube-api-access-d24x6\") pod \"migrator-66df5d496b-sxwqs\" (UID: \"904817ce-d573-4b9e-ba53-646819696439\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"
Oct 30 00:38:31.082576 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.082549    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d24x6\" (UniqueName: \"kubernetes.io/projected/904817ce-d573-4b9e-ba53-646819696439-kube-api-access-d24x6\") pod \"migrator-66df5d496b-sxwqs\" (UID: \"904817ce-d573-4b9e-ba53-646819696439\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"
Oct 30 00:38:31.110660 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.110636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:31.167330 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.167273    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:31.167330 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.167321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:31.167590 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.167524    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:31.183116 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.183082    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:38:31.184261 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.184191    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-44cn9\""
Oct 30 00:38:31.184261 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.184195    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:38:31.184261 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.184220    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:38:31.184459 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.184399    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9xms5\""
Oct 30 00:38:31.184542 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.184525    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-smcj2\""
Oct 30 00:38:31.184602 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.184586    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:38:31.185221 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.185205    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"
Oct 30 00:38:31.347869 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.345182    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9wq8z-zs5sd"]
Oct 30 00:38:31.352854 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:31.352825    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b48a54_79e8_44b4_8d9e_5d1046639884.slice/crio-9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1 WatchSource:0}: Error finding container 9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1: Status 404 returned error can't find the container with id 9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1
Oct 30 00:38:31.367271 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.367008    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:31.367271 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.367054    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:31.367271 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:31.367145    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:38:31.367271 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:31.367217    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.367199805 +0000 UTC m=+32.707327121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : configmap "kube-rbac-proxy" not found
Oct 30 00:38:31.367508 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:31.367306    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:31.367508 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:31.367344    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:32.367331772 +0000 UTC m=+32.707459068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:31.415984 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.415943    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mnj2x" event={"ID":"00bbf453-22c0-42ae-843d-7a9fafe9f7c7","Type":"ContainerStarted","Data":"7160b009a1359edb5160872ba3c9db507ea784b3d33c4b51daaae1b2c5a44f31"}
Oct 30 00:38:31.417222 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.417187    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd" event={"ID":"97b48a54-79e8-44b4-8d9e-5d1046639884","Type":"ContainerStarted","Data":"9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1"}
Oct 30 00:38:31.520101 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.520030    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"]
Oct 30 00:38:31.525887 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:31.525843    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904817ce_d573_4b9e_ba53_646819696439.slice/crio-22925fef216e1c43f2878aebcd1ddfc80459636cf8c28d8fc5b90b6d5793481a WatchSource:0}: Error finding container 22925fef216e1c43f2878aebcd1ddfc80459636cf8c28d8fc5b90b6d5793481a: Status 404 returned error can't find the container with id 22925fef216e1c43f2878aebcd1ddfc80459636cf8c28d8fc5b90b6d5793481a
Oct 30 00:38:31.669728 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.669696    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:31.669899 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:31.669872    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:38:31.669945 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:31.669939    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls podName:1b9f3661-225a-43e3-8f77-5122261f257f nodeName:}" failed. No retries permitted until 2026-10-30 00:38:33.669923649 +0000 UTC m=+34.010050944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls") pod "dns-default-k54rc" (UID: "1b9f3661-225a-43e3-8f77-5122261f257f") : secret "dns-default-metrics-tls" not found
Oct 30 00:38:31.716217 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.716182    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:31.835654 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.835588    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:31.927024 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.925699    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:31.927024 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.925943    2567 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:38:31.945251 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:31.944900    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:32.200614 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.200579    2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jg7w4"]
Oct 30 00:38:32.217136 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.216567    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.226363 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.226270    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jg7w4"]
Oct 30 00:38:32.231640 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.230783    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 00:38:32.231640 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.230798    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 00:38:32.231640 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.230824    2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9lsw5\""
Oct 30 00:38:32.231640 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.230859    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 00:38:32.231640 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.230887    2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 00:38:32.376653 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.376617    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-cabundle\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.376845 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.376678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:32.376845 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.376701    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:32.376845 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.376776    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkhb\" (UniqueName: \"kubernetes.io/projected/6f1f3733-1c4c-492e-9cfd-443848be5859-kube-api-access-4wkhb\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.376845 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.376814    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-key\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.377100 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:32.377008    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:38:32.377100 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:32.377069    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:34.377050182 +0000 UTC m=+34.717177482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : configmap "kube-rbac-proxy" not found
Oct 30 00:38:32.377560 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:32.377537    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:32.377691 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:32.377599    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:34.377583171 +0000 UTC m=+34.717710469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:32.421559 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.421522    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs" event={"ID":"904817ce-d573-4b9e-ba53-646819696439","Type":"ContainerStarted","Data":"22925fef216e1c43f2878aebcd1ddfc80459636cf8c28d8fc5b90b6d5793481a"}
Oct 30 00:38:32.478478 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.478398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-cabundle\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.478478 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.478476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkhb\" (UniqueName: \"kubernetes.io/projected/6f1f3733-1c4c-492e-9cfd-443848be5859-kube-api-access-4wkhb\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.478686 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.478504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-key\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.479419 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.479393    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-cabundle\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.486904 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.486875    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-key\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.503918 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.503873    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkhb\" (UniqueName: \"kubernetes.io/projected/6f1f3733-1c4c-492e-9cfd-443848be5859-kube-api-access-4wkhb\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.541712 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.541674    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:32.586226 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.586199    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:32.743599 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.743520    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:32.890466 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:32.890440    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jg7w4"]
Oct 30 00:38:32.897668 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:32.897637    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1f3733_1c4c_492e_9cfd_443848be5859.slice/crio-288942dd9d3b6fa02f4f63485e2327f3f3a9cf2ddfb9acfc770fff7bc2dfa4ba WatchSource:0}: Error finding container 288942dd9d3b6fa02f4f63485e2327f3f3a9cf2ddfb9acfc770fff7bc2dfa4ba: Status 404 returned error can't find the container with id 288942dd9d3b6fa02f4f63485e2327f3f3a9cf2ddfb9acfc770fff7bc2dfa4ba
Oct 30 00:38:33.389279 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.389130    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:33.427073 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.427031    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs" event={"ID":"904817ce-d573-4b9e-ba53-646819696439","Type":"ContainerStarted","Data":"e4e1149e885d3aba868dd13309f8e1f793ffacc46e126c7ed165f712ce234c4c"}
Oct 30 00:38:33.428843 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.428813    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4" event={"ID":"6f1f3733-1c4c-492e-9cfd-443848be5859","Type":"ContainerStarted","Data":"288942dd9d3b6fa02f4f63485e2327f3f3a9cf2ddfb9acfc770fff7bc2dfa4ba"}
Oct 30 00:38:33.475698 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.475642    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs" podStartSLOduration=2.117508945 podStartE2EDuration="3.475626372s" totalImagesPullingTime="1.358117427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:33.475101371 +0000 UTC m=+33.815228691" watchObservedRunningTime="2026-10-30 00:38:33.475626372 +0000 UTC m=+33.815753680"
Oct 30 00:38:33.689472 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.689433    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:33.689639 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:33.689600    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:38:33.689692 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:33.689677    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls podName:1b9f3661-225a-43e3-8f77-5122261f257f nodeName:}" failed. No retries permitted until 2026-10-30 00:38:37.68965724 +0000 UTC m=+38.029784541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls") pod "dns-default-k54rc" (UID: "1b9f3661-225a-43e3-8f77-5122261f257f") : secret "dns-default-metrics-tls" not found
Oct 30 00:38:33.891093 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.891056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:33.891297 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.891165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:33.891367 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:33.891305    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:38:33.891406 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:33.891372    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:39:05.891354081 +0000 UTC m=+66.231481391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : secret "metrics-daemon-secret" not found
Oct 30 00:38:33.893939 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.893907    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:33.915948 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.915920    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:33.991854 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.991831    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:33.994731 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:33.994709    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:34.236691 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.236600    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:34.301995 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.301947    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4h2rk"]
Oct 30 00:38:34.309397 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:34.309183    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a648c9_5e85_49b8_91f3_967fad2836f8.slice/crio-baa64a736c629a4a437aff098e9da447da2aa7fd991ed4a20579f1f54b737495 WatchSource:0}: Error finding container baa64a736c629a4a437aff098e9da447da2aa7fd991ed4a20579f1f54b737495: Status 404 returned error can't find the container with id baa64a736c629a4a437aff098e9da447da2aa7fd991ed4a20579f1f54b737495
Oct 30 00:38:34.395869 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.395833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:34.395869 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.395872    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:34.396379 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:34.395988    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:34.396379 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:34.395994    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:38:34.396379 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:34.396041    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:38.396028451 +0000 UTC m=+38.736155746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:34.396379 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:34.396067    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:38.396047012 +0000 UTC m=+38.736174323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : configmap "kube-rbac-proxy" not found
Oct 30 00:38:34.433128 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.433095    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mnj2x" event={"ID":"00bbf453-22c0-42ae-843d-7a9fafe9f7c7","Type":"ContainerStarted","Data":"8e8681b70cc55764e93ac1266b2d0e78d30567b80a68556dcf96eb665981ab4a"}
Oct 30 00:38:34.434474 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.434446    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4h2rk" event={"ID":"56a648c9-5e85-49b8-91f3-967fad2836f8","Type":"ContainerStarted","Data":"baa64a736c629a4a437aff098e9da447da2aa7fd991ed4a20579f1f54b737495"}
Oct 30 00:38:34.436171 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.436141    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd" event={"ID":"97b48a54-79e8-44b4-8d9e-5d1046639884","Type":"ContainerStarted","Data":"3c6b081cf99805a5a0ab50fd0550bf4da7e92585cd0dada73ce169e23c112ad6"}
Oct 30 00:38:34.472364 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.472316    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mnj2x" podStartSLOduration=2.217666377 podStartE2EDuration="5.472302691s" totalImagesPullingTime="3.254636314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:34.472043584 +0000 UTC m=+34.812170903" watchObservedRunningTime="2026-10-30 00:38:34.472302691 +0000 UTC m=+34.812430010"
Oct 30 00:38:34.509220 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.509133    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd" podStartSLOduration=1.8466498489999998 podStartE2EDuration="4.509119361s" totalImagesPullingTime="2.662469512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:34.508500389 +0000 UTC m=+34.848627707" watchObservedRunningTime="2026-10-30 00:38:34.509119361 +0000 UTC m=+34.849246678"
Oct 30 00:38:34.549833 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.549803    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tds5t"]
Oct 30 00:38:34.554620 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:34.554590    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40aff960_4844_46db_aa92_2efb92303076.slice/crio-ff70604a5d0d291631886fdc5613ebc64bad59cf9c6989e124212e3dbe4ec061 WatchSource:0}: Error finding container ff70604a5d0d291631886fdc5613ebc64bad59cf9c6989e124212e3dbe4ec061: Status 404 returned error can't find the container with id ff70604a5d0d291631886fdc5613ebc64bad59cf9c6989e124212e3dbe4ec061
Oct 30 00:38:34.695902 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.695866    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:34.849378 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:34.849290    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:35.440377 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:35.440302    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tds5t" event={"ID":"40aff960-4844-46db-aa92-2efb92303076","Type":"ContainerStarted","Data":"ff70604a5d0d291631886fdc5613ebc64bad59cf9c6989e124212e3dbe4ec061"}
Oct 30 00:38:36.052760 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:36.052683    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:36.445375 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:36.445328    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4" event={"ID":"6f1f3733-1c4c-492e-9cfd-443848be5859","Type":"ContainerStarted","Data":"86cbec58e7221691ac29f9bb96186ad83cd610294a400020c85d63c2e07216fa"}
Oct 30 00:38:36.455652 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:36.455619    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:36.594995 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:36.594966    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:36.743344 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:36.743266    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6j4cc_edfab0af-8070-4ffb-9c66-9c94399393d7/dns-node-resolver/0.log"
Oct 30 00:38:37.126380 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:37.126181    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:37.349272 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:37.349203    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:37.478249 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:37.478163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:37.726834 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:37.726790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:37.732216 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:37.732112    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:37.852746 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:37.852704    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:37.943534 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:37.943505    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-frssv_38381a16-212b-4d9c-98c2-6d81e4af3b74/node-ca/0.log"
Oct 30 00:38:38.018091 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.018018    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:38.323045 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.322956    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4" podStartSLOduration=3.506545496 podStartE2EDuration="6.322939928s" totalImagesPullingTime="2.816394432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:36.486467344 +0000 UTC m=+36.826594666" watchObservedRunningTime="2026-10-30 00:38:38.322939928 +0000 UTC m=+38.663067246"
Oct 30 00:38:38.323556 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.323539    2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k54rc"]
Oct 30 00:38:38.328544 ip-10-0-130-5 kubenswrapper[2567]: W1030 00:38:38.328516    2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9f3661_225a_43e3_8f77_5122261f257f.slice/crio-9e57ed1b798ee376f246a060274d760f60e8f45e005a98e45c4eb57176584691 WatchSource:0}: Error finding container 9e57ed1b798ee376f246a060274d760f60e8f45e005a98e45c4eb57176584691: Status 404 returned error can't find the container with id 9e57ed1b798ee376f246a060274d760f60e8f45e005a98e45c4eb57176584691
Oct 30 00:38:38.433702 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.433669    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:38.433823 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.433721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:38.433823 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:38.433811    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:38:38.433931 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:38.433838    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:38.433931 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:38.433876    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:46.433860696 +0000 UTC m=+46.773987996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : configmap "kube-rbac-proxy" not found
Oct 30 00:38:38.433931 ip-10-0-130-5 kubenswrapper[2567]: E1030 00:38:38.433890    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:46.433884781 +0000 UTC m=+46.774012077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:38.456186 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.456139    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tds5t" event={"ID":"40aff960-4844-46db-aa92-2efb92303076","Type":"ContainerStarted","Data":"e5388510a5ea8f8185d94be3c6f7dcf3cf60e8f888879066d70c65da10619746"}
Oct 30 00:38:38.456363 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.456272    2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:38.457587 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.457553    2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k54rc" event={"ID":"1b9f3661-225a-43e3-8f77-5122261f257f","Type":"ContainerStarted","Data":"9e57ed1b798ee376f246a060274d760f60e8f45e005a98e45c4eb57176584691"}
Oct 30 00:38:38.495389 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.495329    2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tds5t" podStartSLOduration=34.712449074 podStartE2EDuration="38.495309131s" totalImagesPullingTime="3.782860057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:38.49444768 +0000 UTC m=+38.834575012" watchObservedRunningTime="2026-10-30 00:38:38.495309131 +0000 UTC m=+38.835436452"
Oct 30 00:38:38.649169 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.649077    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:38.749822 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:38.749787    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mnj2x_00bbf453-22c0-42ae-843d-7a9fafe9f7c7/serve-healthcheck-canary/0.log"
Oct 30 00:38:39.004460 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:39.004428    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:39.125792 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:39.125763    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:39.143472 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:39.143446    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-sxwqs_904817ce-d573-4b9e-ba53-646819696439/migrator/0.log"
Oct 30 00:38:39.272025 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:39.271950    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:39.441985 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:39.441946    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:39.846658 ip-10-0-130-5 kubenswrapper[2567]: I1030 00:38:39.846581    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:39.861398 ip-10-0-130-5 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 00:38:39.882732 ip-10-0-130-5 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 00:38:39.882975 ip-10-0-130-5 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 00:38:39.883037 ip-10-0-130-5 systemd[1]: kubelet.service: Consumed 2.509s CPU time.
Oct 30 00:38:39.892875 ip-10-0-130-5 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:38:39.972909 ip-10-0-130-5 kubenswrapper[4576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:38:39.972909 ip-10-0-130-5 kubenswrapper[4576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:38:39.972909 ip-10-0-130-5 kubenswrapper[4576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:38:39.972909 ip-10-0-130-5 kubenswrapper[4576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:38:39.976246 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976125    4576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:38:39.976246 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976249    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:38:39.976246 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976255    4576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976261    4576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976266    4576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976271    4576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976275    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976285    4576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976290    4576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976294    4576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976298    4576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976303    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976307    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976311    4576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976315    4576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976319    4576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976325    4576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976329    4576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976333    4576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976337    4576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976341    4576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976346    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:38:39.976425 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976350    4576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976354    4576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976358    4576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976363    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976367    4576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976371    4576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976376    4576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976380    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976384    4576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976388    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976392    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976396    4576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976400    4576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976405    4576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976409    4576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976413    4576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976417    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976422    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976426    4576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:38:39.977072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976430    4576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976435    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976439    4576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976443    4576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976447    4576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976451    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976455    4576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976459    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976464    4576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976468    4576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976471    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976476    4576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976480    4576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976483    4576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976488    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976492    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976499    4576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976508    4576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976513    4576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:38:39.977685 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976517    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976522    4576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976526    4576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976530    4576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976534    4576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976539    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976543    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976547    4576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976551    4576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976556    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976559    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976563    4576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976567    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976572    4576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976576    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976580    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976584    4576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976588    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976592    4576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976596    4576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:38:39.978282 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976600    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976604    4576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976612    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976616    4576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976620    4576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976624    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976627    4576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976631    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976635    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976639    4576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976644    4576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976648    4576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976652    4576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976656    4576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976660    4576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976664    4576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976668    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976672    4576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976677    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:38:39.978894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976681    4576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976687    4576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976695    4576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976699    4576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976703    4576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976707    4576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.976712    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977912    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977920    4576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977925    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977929    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977934    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977938    4576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977944    4576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977951    4576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977957    4576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977962    4576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977966    4576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977972    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:38:39.979490 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977976    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977980    4576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977985    4576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977990    4576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977995    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.977999    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978005    4576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978010    4576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978014    4576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978019    4576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978023    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978027    4576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978032    4576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978037    4576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978042    4576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978046    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978050    4576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978054    4576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978058    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:38:39.980014 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978062    4576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978066    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978071    4576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978075    4576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978079    4576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978082    4576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978087    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978091    4576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978095    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978099    4576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978103    4576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978107    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978111    4576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978115    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978119    4576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978123    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978128    4576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978134    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978138    4576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:38:39.980632 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978142    4576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978146    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978150    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978154    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978157    4576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978161    4576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978165    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978169    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978174    4576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978178    4576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978182    4576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978185    4576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978190    4576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978194    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978198    4576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978202    4576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978206    4576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978210    4576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978214    4576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978218    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:38:39.981240 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978222    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978227    4576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978247    4576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978251    4576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978256    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978260    4576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978264    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978268    4576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978272    4576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978276    4576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978280    4576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978284    4576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978289    4576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978293    4576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978297    4576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978302    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978306    4576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978310    4576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978314    4576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978318    4576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:38:39.981894 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978322    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978327    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978331    4576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978335    4576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978339    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978343    4576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978347    4576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978351    4576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978355    4576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978360    4576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978363    4576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978372    4576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978378    4576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978384    4576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978389    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.978394    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979126    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979133    4576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979137    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979141    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:38:39.982532 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979146    4576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979150    4576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979154    4576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979159    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979163    4576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979167    4576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979173    4576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979177    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979181    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979185    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979189    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979194    4576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979198    4576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979202    4576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979207    4576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979211    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979215    4576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979219    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979223    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979227    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:38:39.983316 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979247    4576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979251    4576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979256    4576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979261    4576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979265    4576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979269    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979273    4576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979277    4576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979281    4576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979285    4576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979292    4576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979297    4576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979302    4576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979306    4576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979311    4576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979315    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979318    4576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979323    4576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979328    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:38:39.984005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979333    4576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979337    4576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979341    4576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979345    4576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979350    4576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979354    4576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979358    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979363    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979367    4576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979372    4576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979376    4576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979380    4576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979384    4576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979387    4576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979391    4576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979395    4576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979400    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979404    4576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979408    4576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979412    4576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:38:39.984947 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979418    4576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979422    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979426    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979429    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979433    4576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979438    4576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979441    4576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979446    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979450    4576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979453    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979457    4576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979463    4576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979467    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979471    4576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979475    4576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979480    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979484    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979488    4576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979492    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979496    4576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:38:39.985775 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979501    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979505    4576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979510    4576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979514    4576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979518    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979522    4576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979526    4576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979530    4576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979535    4576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979539    4576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979543    4576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979547    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979551    4576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979556    4576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979560    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979564    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979568    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979574    4576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979581    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:38:39.986641 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979587    4576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979592    4576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979596    4576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.979601    4576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979734    4576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979744    4576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979753    4576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979761    4576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979767    4576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979773    4576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979779    4576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979785    4576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979790    4576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979795    4576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979801    4576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979806    4576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979811    4576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979815    4576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979820    4576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979824    4576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979829    4576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979833    4576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979839    4576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:38:39.987458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979844    4576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979848    4576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979853    4576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979858    4576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979864    4576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979869    4576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979874    4576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979879    4576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979883    4576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979888    4576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979892    4576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979897    4576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979902    4576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979909    4576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979914    4576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979919    4576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979923    4576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979928    4576 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979932    4576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979942    4576 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979949    4576 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979954    4576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979958    4576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979963    4576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:38:39.988416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979970    4576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979975    4576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979980    4576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979984    4576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979989    4576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979994    4576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.979999    4576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980003    4576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980007    4576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980012    4576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980016    4576 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980023    4576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980027    4576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980032    4576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980037    4576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980042    4576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980046    4576 flags.go:64] FLAG: --help="false"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980051    4576 flags.go:64] FLAG: --hostname-override="ip-10-0-130-5.ec2.internal"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980056    4576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980061    4576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980066    4576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980072    4576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980077    4576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980081    4576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:38:39.989045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980086    4576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980090    4576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980095    4576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980099    4576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980105    4576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980110    4576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980115    4576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980119    4576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980124    4576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980129    4576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980133    4576 flags.go:64] FLAG: --lock-file=""
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980138    4576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980142    4576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980147    4576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980155    4576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980160    4576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980164    4576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980169    4576 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980173    4576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980178    4576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980183    4576 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980187    4576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980194    4576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980198    4576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980204    4576 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:38:39.989702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980209    4576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980213    4576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980218    4576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980223    4576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980227    4576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980248    4576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980252    4576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980263    4576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980267    4576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980272    4576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980277    4576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980282    4576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980287    4576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980292    4576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980296    4576 flags.go:64] FLAG: --port="10250"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980301    4576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980306    4576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-062ddcd3c54899bb9"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980311    4576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980316    4576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980321    4576 flags.go:64] FLAG: --register-node="true"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980325    4576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980331    4576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980335    4576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980340    4576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980344    4576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980350    4576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:38:39.990524 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980354    4576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980359    4576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980363    4576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980368    4576 flags.go:64] FLAG: --runonce="false"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980373    4576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980377    4576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980382    4576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980387    4576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980392    4576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980396    4576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980401    4576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980406    4576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980411    4576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980415    4576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980420    4576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980424    4576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980429    4576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980435    4576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980439    4576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980447    4576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980451    4576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980455    4576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980461    4576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980466    4576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980471    4576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:38:39.991772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980476    4576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980480    4576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980485    4576 flags.go:64] FLAG: --v="2"
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980491    4576 flags.go:64] FLAG: --version="false"
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980497    4576 flags.go:64] FLAG: --vmodule=""
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980503    4576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.980508    4576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980743    4576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980748    4576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980753    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980757    4576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980761    4576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980764    4576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980768    4576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980773    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980777    4576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980781    4576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980785    4576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980788    4576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980793    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980797    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:38:39.993034 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980801    4576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980805    4576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980810    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980816    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980820    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980824    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980828    4576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980835    4576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980841    4576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980854    4576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980859    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980866    4576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980870    4576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980875    4576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980879    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980883    4576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980887    4576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980891    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980895    4576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:38:39.993934 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980899    4576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980903    4576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980907    4576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980911    4576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980916    4576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980922    4576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980927    4576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980931    4576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980936    4576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980940    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980944    4576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980948    4576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980952    4576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980956    4576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980960    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980966    4576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980972    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980976    4576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980980    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:38:39.994793 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980984    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980988    4576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980992    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.980997    4576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981001    4576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981005    4576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981010    4576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981014    4576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981018    4576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981022    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981026    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981030    4576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981035    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981039    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981043    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981047    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981051    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981055    4576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981059    4576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:38:39.995615 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981063    4576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981067    4576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981071    4576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981075    4576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981079    4576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981083    4576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981087    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981091    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981095    4576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981101    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981106    4576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981110    4576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981114    4576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981118    4576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981123    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981127    4576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981131    4576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981135    4576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981139    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981143    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:38:39.996529 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981150    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981155    4576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981159    4576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981163    4576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981168    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981172    4576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981176    4576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981180    4576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981184    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981188    4576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981192    4576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981195    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981200    4576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981204    4576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981208    4576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.981262    4576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:38:39.997391 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981323    4576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981340    4576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981345    4576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981349    4576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981353    4576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981359    4576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981365    4576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981373    4576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981378    4576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981383    4576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981387    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981391    4576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981395    4576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981399    4576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981403    4576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981407    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981411    4576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981423    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981428    4576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:38:39.998052 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981432    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981436    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981440    4576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981444    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981448    4576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981452    4576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981456    4576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981460    4576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981464    4576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981469    4576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981473    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981477    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981481    4576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981485    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981489    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981493    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981497    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981501    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981515    4576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:38:39.998861 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981520    4576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981526    4576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981530    4576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981534    4576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981538    4576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981542    4576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981546    4576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981550    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981554    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981558    4576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981562    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981567    4576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981582    4576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981586    4576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981590    4576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981594    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981599    4576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981602    4576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981607    4576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981611    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:38:39.999699 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981615    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981619    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981623    4576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981627    4576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981631    4576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981635    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981639    4576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981643    4576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981647    4576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981651    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981655    4576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981661    4576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981665    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981670    4576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981674    4576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981678    4576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981682    4576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981686    4576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981690    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981695    4576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:38:40.000556 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981699    4576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981703    4576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981707    4576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981711    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981716    4576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981726    4576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981730    4576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981734    4576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981738    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981742    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981746    4576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981750    4576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981754    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981758    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981762    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981766    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981770    4576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981778    4576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981784    4576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:38:40.001417 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981788    4576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981793    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981797    4576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981801    4576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981806    4576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981811    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981816    4576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981820    4576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.981824    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.987399    4576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.987417    4576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.987435    4576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.987440    4576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987509    4576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987518    4576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987523    4576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987528    4576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987533    4576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987538    4576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987543    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:38:40.002249 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987548    4576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987553    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987557    4576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987560    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987565    4576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987569    4576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987573    4576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987576    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987578    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987581    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987583    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987586    4576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987589    4576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987591    4576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987593    4576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987596    4576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987599    4576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987603    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987605    4576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987608    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:38:40.003068 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987610    4576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987613    4576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987627    4576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987631    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987633    4576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987637    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987642    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987647    4576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987651    4576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987653    4576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987656    4576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987659    4576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987662    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987664    4576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987667    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987670    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987672    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987674    4576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987677    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:38:40.003941 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987679    4576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987682    4576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987684    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987687    4576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987689    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987691    4576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987698    4576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987704    4576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987709    4576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987713    4576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987718    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987722    4576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987727    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987730    4576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987733    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987735    4576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987744    4576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987747    4576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987750    4576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:38:40.004690 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987752    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987755    4576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987758    4576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987760    4576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987763    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987766    4576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987768    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987773    4576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987776    4576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987779    4576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987782    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987785    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987787    4576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987789    4576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987793    4576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987796    4576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987799    4576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987801    4576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987804    4576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:38:40.005462 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987807    4576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987810    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987812    4576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987814    4576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987817    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987820    4576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987822    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987825    4576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987828    4576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987832    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987836    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987848    4576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987851    4576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987854    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987858    4576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987863    4576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987867    4576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987870    4576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987873    4576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:38:40.006275 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987876    4576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987878    4576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.987881    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.987919    4576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988834    4576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988851    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988857    4576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988867    4576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988872    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988877    4576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988881    4576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988886    4576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988891    4576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988895    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988900    4576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988904    4576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:38:40.007028 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988908    4576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988912    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988917    4576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988921    4576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988931    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988935    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988939    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988943    4576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988947    4576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988975    4576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988980    4576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988985    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988989    4576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988993    4576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.988997    4576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989001    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989011    4576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989015    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989020    4576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989025    4576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:38:40.007667 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989029    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989034    4576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989038    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989042    4576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989047    4576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989052    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989056    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989061    4576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989065    4576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989074    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989079    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989084    4576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989088    4576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989094    4576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989101    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989106    4576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989111    4576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989115    4576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989119    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:38:40.008476 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989124    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989128    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989137    4576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989165    4576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989170    4576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989174    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989178    4576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989182    4576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989186    4576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989190    4576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989194    4576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989198    4576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989202    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989206    4576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989211    4576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989220    4576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989225    4576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989245    4576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989250    4576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989255    4576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:38:40.009216 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989259    4576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989263    4576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989267    4576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989272    4576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989276    4576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989280    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989284    4576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989288    4576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989297    4576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989302    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989306    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989311    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989315    4576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989319    4576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989324    4576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989328    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989363    4576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989369    4576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989373    4576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:38:40.010035 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989377    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989388    4576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989395    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989399    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989404    4576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989408    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989412    4576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989416    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989421    4576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989425    4576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989429    4576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989434    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989438    4576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989442    4576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989451    4576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:38:40.010811 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989455    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.989516    4576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989763    4576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989783    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989788    4576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989792    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989799    4576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989802    4576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989805    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989808    4576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989811    4576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989814    4576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989851    4576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989856    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989860    4576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989863    4576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:38:40.011441 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989865    4576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989868    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989871    4576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989874    4576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989878    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989880    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989883    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989886    4576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989892    4576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989894    4576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989897    4576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989901    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989904    4576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989907    4576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989909    4576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989912    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989915    4576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989919    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989923    4576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989935    4576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:38:40.012076 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989939    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989944    4576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989947    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989950    4576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989953    4576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989956    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989959    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989962    4576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989964    4576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989967    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989970    4576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989973    4576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989976    4576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989980    4576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989993    4576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989996    4576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.989999    4576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990002    4576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990005    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:38:40.012871 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990008    4576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990011    4576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990014    4576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990017    4576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990027    4576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990030    4576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990036    4576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990039    4576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990041    4576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990044    4576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990047    4576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990050    4576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990052    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990061    4576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990064    4576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990066    4576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990069    4576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990072    4576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990074    4576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990079    4576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:38:40.013728 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990082    4576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990085    4576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990087    4576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990090    4576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990092    4576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990095    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990097    4576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990100    4576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990103    4576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990106    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990108    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990111    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990116    4576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990122    4576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990124    4576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990127    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990130    4576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990133    4576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990135    4576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:38:40.014534 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990138    4576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990147    4576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990151    4576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990160    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990167    4576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990170    4576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990172    4576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990181    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990184    4576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990187    4576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990190    4576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990192    4576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990195    4576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:39.990198    4576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.990505    4576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.991607    4576 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.991684    4576 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.991901    4576 server.go:1014] "Starting client certificate rotation"
Oct 30 00:38:40.015288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.992087    4576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:38:40.016064 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.992145    4576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:33:00 +0000 UTC" deadline="2028-06-03 20:46:50.720351667 +0000 UTC"
Oct 30 00:38:40.016064 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.992191    4576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13988h8m10.728162749s"
Oct 30 00:38:40.016064 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:39.992497    4576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:38:40.021992 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.021960    4576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:38:40.023968 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.023936    4576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:38:40.026001 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.025985    4576 log.go:25] "Validated CRI v1 image API"
Oct 30 00:38:40.027102 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.027048    4576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:38:40.029459 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.029429    4576 fs.go:135] Filesystem UUIDs: map[2b6d4360-30aa-4302-98ac-683b42a7b635:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d83562f5-ef61-4b9d-97b5-257dcd3a6603:/dev/nvme0n1p4]
Oct 30 00:38:40.029735 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.029460    4576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13b536f0aead184c905ca8c26683444e28599553e9300b6892080c9df3f63ae8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13b536f0aead184c905ca8c26683444e28599553e9300b6892080c9df3f63ae8/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1455a5657ba343086772ff987a745481368d49e09c28041761155497e7398c03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1455a5657ba343086772ff987a745481368d49e09c28041761155497e7398c03/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22925fef216e1c43f2878aebcd1ddfc80459636cf8c28d8fc5b90b6d5793481a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22925fef216e1c43f2878aebcd1ddfc80459636cf8c28d8fc5b90b6d5793481a/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/240f181228572cf3999d7ee380753ddfb4bc862a5d5ad6b5d7471023e9e9ca33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/240f181228572cf3999d7ee380753ddfb4bc862a5d5ad6b5d7471023e9e9ca33/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/288942dd9d3b6fa02f4f63485e2327f3f3a9cf2ddfb9acfc770fff7bc2dfa4ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/288942dd9d3b6fa02f4f63485e2327f3f3a9cf2ddfb9acfc770fff7bc2dfa4ba/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a92bd22db7888c525b0fc350f89d7199dc33679ae842a52b2399800919c7dd7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a92bd22db7888c525b0fc350f89d7199dc33679ae842a52b2399800919c7dd7/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7160b009a1359edb5160872ba3c9db507ea784b3d33c4b51daaae1b2c5a44f31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7160b009a1359edb5160872ba3c9db507ea784b3d33c4b51daaae1b2c5a44f31/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/775233877f80c24f544008201c7230151e8f583c1c7df2598c9865dbd3245ec6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/775233877f80c24f544008201c7230151e8f583c1c7df2598c9865dbd3245ec6/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e57ed1b798ee376f246a060274d760f60e8f45e005a98e45c4eb57176584691/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e57ed1b798ee376f246a060274d760f60e8f45e005a98e45c4eb57176584691/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aab0c2e8f9f7615a708f0505d039b5a06854eb26daa49c0ef0ffd01191a192b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aab0c2e8f9f7615a708f0505d039b5a06854eb26daa49c0ef0ffd01191a192b0/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b42d909c9e24bea099fed03d603ebccf4a27aa47b91d5a5161f301b7fa5d8225/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b42d909c9e24bea099fed03d603ebccf4a27aa47b91d5a5161f301b7fa5d8225/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/baa64a736c629a4a437aff098e9da447da2aa7fd991ed4a20579f1f54b737495/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/baa64a736c629a4a437aff098e9da447da2aa7fd991ed4a20579f1f54b737495/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3fa82872f31f5f151ea52ec1d18a3e46dd820c4dd7012960311453bdba8eecf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3fa82872f31f5f151ea52ec1d18a3e46dd820c4dd7012960311453bdba8eecf/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1fd7b895496783c7254d47d97bb48f11a7f1e4ea4a83241f9ac89baeb2965b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1fd7b895496783c7254d47d97bb48f11a7f1e4ea4a83241f9ac89baeb2965b0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2720d61591632d53e72a27378c6c0a955158c1c54bbd6aaed6f4deda9b5feaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2720d61591632d53e72a27378c6c0a955158c1c54bbd6aaed6f4deda9b5feaf/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff70604a5d0d291631886fdc5613ebc64bad59cf9c6989e124212e3dbe4ec061/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff70604a5d0d291631886fdc5613ebc64bad59cf9c6989e124212e3dbe4ec061/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00bbf453-22c0-42ae-843d-7a9fafe9f7c7/volumes/kubernetes.io~projected/kube-api-access-82s67:{mountpoint:/var/lib/kubelet/pods/00bbf453-22c0-42ae-843d-7a9fafe9f7c7/volumes/kubernetes.io~projected/kube-api-access-82s67 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00bbf453-22c0-42ae-843d-7a9fafe9f7c7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/00bbf453-22c0-42ae-843d-7a9fafe9f7c7/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~projected/kube-api-access-j99fx:{mountpoint:/var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~projected/kube-api-access-j99fx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b9f3661-225a-43e3-8f77-5122261f257f/volumes/kubernetes.io~projected/kube-api-access-49br2:{mountpoint:/var/lib/kubelet/pods/1b9f3661-225a-43e3-8f77-5122261f257f/volumes/kubernetes.io~projected/kube-api-access-49br2 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b9f3661-225a-43e3-8f77-5122261f257f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1b9f3661-225a-43e3-8f77-5122261f257f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38381a16-212b-4d9c-98c2-6d81e4af3b74/volumes/kubernetes.io~projected/kube-api-access-vlfgn:{mountpoint:/var/lib/kubelet/pods/38381a16-212b-4d9c-98c2-6d81e4af3b74/volumes/kubernetes.io~projected/kube-api-access-vlfgn major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bf88444-b628-4aba-a61e-82ae03285b9e/volumes/kubernetes.io~projected/kube-api-access-5fcdt:{mountpoint:/var/lib/kubelet/pods/3bf88444-b628-4aba-a61e-82ae03285b9e/volumes/kubernetes.io~projected/kube-api-access-5fcdt major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40aff960-4844-46db-aa92-2efb92303076/volumes/kubernetes.io~projected/kube-api-access-d6zxt:{mountpoint:/var/lib/kubelet/pods/40aff960-4844-46db-aa92-2efb92303076/volumes/kubernetes.io~projected/kube-api-access-d6zxt major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56a648c9-5e85-49b8-91f3-967fad2836f8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/56a648c9-5e85-49b8-91f3-967fad2836f8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f1f3733-1c4c-492e-9cfd-443848be5859/volumes/kubernetes.io~projected/kube-api-access-4wkhb:{mountpoint:/var/lib/kubelet/pods/6f1f3733-1c4c-492e-9cfd-443848be5859/volumes/kubernetes.io~projected/kube-api-access-4wkhb major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f1f3733-1c4c-492e-9cfd-443848be5859/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/6f1f3733-1c4c-492e-9cfd-443848be5859/volumes/kubernetes.io~secret/signing-key major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87032b63-645e-4392-bd02-80b1289399c3/volumes/kubernetes.io~projected/kube-api-access-2hlvr:{mountpoint:/var/lib/kubelet/pods/87032b63-645e-4392-bd02-80b1289399c3/volumes/kubernetes.io~projected/kube-api-access-2hlvr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/904817ce-d573-4b9e-ba53-646819696439/volumes/kubernetes.io~projected/kube-api-access-d24x6:{mountpoint:/var/lib/kubelet/pods/904817ce-d573-4b9e-ba53-646819696439/volumes/kubernetes.io~projected/kube-api-access-d24x6 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97b48a54-79e8-44b4-8d9e-5d1046639884/volumes/kubernetes.io~projected/kube-api-access-552tl:{mountpoint:/var/lib/kubelet/pods/97b48a54-79e8-44b4-8d9e-5d1046639884/volumes/kubernetes.io~projected/kube-api-access-552tl major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b899a78-408a-433c-8e45-c7dd7499a1f2/volumes/kubernetes.io~projected/kube-api-access-ttgzz:{mountpoint:/var/lib/kubelet/pods/9b899a78-408a-433c-8e45-c7dd7499a1f2/volumes/kubernetes.io~projected/kube-api-access-ttgzz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a312a1ca-eba6-4a80-b070-caee6de84c04/volumes/kubernetes.io~projected/kube-api-access-tld2q:{mountpoint:/var/lib/kubelet/pods/a312a1ca-eba6-4a80-b070-caee6de84c04/volumes/kubernetes.io~projected/kube-api-access-tld2q major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7f726d8-6d17-4370-a98c-41a979b549ae/volumes/kubernetes.io~projected/kube-api-access-69lpl:{mountpoint:/var/lib/kubelet/pods/c7f726d8-6d17-4370-a98c-41a979b549ae/volumes/kubernetes.io~projected/kube-api-access-69lpl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10c0ce2-a04c-4f4f-9977-db48be2e5504/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d10c0ce2-a04c-4f4f-9977-db48be2e5504/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edfab0af-8070-4ffb-9c66-9c94399393d7/volumes/kubernetes.io~projected/kube-api-access-8zv44:{mountpoint:/var/lib/kubelet/pods/edfab0af-8070-4ffb-9c66-9c94399393d7/volumes/kubernetes.io~projected/kube-api-access-8zv44 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1d925dd-7561-4ee2-8e42-f38ee71296f6/volumes/kubernetes.io~projected/kube-api-access-7l6sm:{mountpoint:/var/lib/kubelet/pods/f1d925dd-7561-4ee2-8e42-f38ee71296f6/volumes/kubernetes.io~projected/kube-api-access-7l6sm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volumes/kubernetes.io~projected/kube-api-access-rbvph:{mountpoint:/var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volumes/kubernetes.io~projected/kube-api-access-rbvph major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d8b856ec9129525fe4c4a0902f0cdf73ed9e4a0f94822ada18618df171fd7367/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4edce9c917ce42029bbf93075aeae40eb9d1573310eefbe09a3df59ed6273ace/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/611c147dfdfa584c6830bdec933d9fe5a9525764130deaf4c8231cfcac8ad7c1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/766727bb49ead18562cfc26daa26776b674e8b2c3f9c6050b8834cd9c3d492ee/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9dce3dd28998e79edb3e70db51ea7f3b7078b9f6ee28654c6a89cde9351ef444/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9ec436c2fc3f7c9382bfeedd9f716f717bdcb1c802ac82373d590330ad6a4586/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/834157b35689b9decff2f2307f57432986b7369ed31955549e5fba5373f12760/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/144d73fdcbe9835115c3fea04c4522f8fd18c4c7667b8a8ebb87c36227ab04a8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/cc70aae79268af129b5c3a31198f0b48cc248159de877e64b267e7b9f5343499/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f54c650b5b09ec52feb3aded56769d33899be696baac9fa25778ec707a7185cd/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/893b646be549defd2206b00895fe27e6c76ebcf2dc3fd3b2b924b15dc99e06e0/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/c1eec44fce82d5ecddb3b3ded5de69269611cb2d331d692258ea0cf966943f95/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a22f62609da92464a6e12cfa17de91fdbde0c6c419bebcc648b314190c163a8f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/440d1a95c4f6a85c3dbb7a80019d4685d373e7068298fa3f0bb113c236a1f012/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a48eefea00d916466a9fdf6d90a1fb25fbdc166375af7accf0a8dc75fc1030ca/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/437dda20079d783c81582cecb94ffbf85302e7420b9bca35cda9526eb4686fd1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/881cb116000a304bb52abebd090c10191b581bfe6dae2974824eb0e08235bc7d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/98ea7944c9685266e3d708dca3621ae59d44f6ba206e5f5b3ce34ab076d87ef5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/52e2c8693ee90a3e21d04efda706c41384fe8197429dbe71968c265b896cea35/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cf765293919d41abb1a949aec83f07f3958db8b100a7e0d8ee956748485fc736/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ba0503117b254e757274b32a001d57bb45747322b522543bacc65fbbab551b71/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/7e69faf4cfd3077e44692cd24de377d9e63d913502e265efa5be5fca439cd2de/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/7268c5767803cadbd7b05f3d9972244b5ef0c9b4b969428559b0abf8dea7038d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/1cfae468ac6c3817fc24894b581bb195b7132c8cfeaf6797fd2581e2c98e148e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/63567774a3d101ef74e10af76c32001f37774a7d631f4d46751c93b41568a4d7/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/066325d0576d56e2b97f43231f4dc68d081797ef9c36e5ab6b378da9b5b31f00/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4c0abf417d15f857e133380c43351bc8f798acafe73c9e14fcd38b78c0b1ef57/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/c492bea159596a2a74436ad7eb1916d03b7ed8f233b2d6c5b48f8614f15cadeb/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/27eefc55754f08bc478042848dd039e7f336f727da2265738021affc7c3fe5fa/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/28bb3f5efd1856c3ec87d93c8835b8b52977f50c6e06d6586994c31ca4549725/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/6b9b8932e3466b261b07249ccc7360c30f9972fba2c5fce5ed37eb8fc0f885b4/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/2b0c015a6d3b57faf4eccbfffdd35a788ca395fd3e0002e291d9d5cd833af0dd/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f16607cc9aa1bb6c53ed6adf47e5915b45662dd90fff18dab887b6b477abe104/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/efd6fb0d835764de2a1067b7058486c17c457d2b6388f46ca95c2f6fa69d5b4b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2e195a25fa07fdeaef37fb656fd94b10ea78c4607b8b33ad767dea1a769b0cd8/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c8825b996fe88500908f86b58d7fba41efac26780de9c0750522145bb3c93873/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/6f7f1238d726d398d46bd9a1eca4d3605f6b8abad0c31cee134f27953aa9c80f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/ded733051e027c1806c0cd5ada5c3766224b4315305527985276f38d633a6ecc/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/7b746f834243f2de5604a4c3eef32d9c89712ceb5e4c2e2a11192d034bf7b76c/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e59d864dbe049cd22b4fba54d30ccdc93c52f3d7f6afe0d9e89c01bb496185b4/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a53c9a155bf1f8723ce30988bd6774b14544f147c3db6d6ce695d530da8daf11/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/08bbbb4e5cf1a08f4eb0bacb43bd504fd1b849a5b2118e69b30d1538bd49f1c1/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/cf3b25a1095ec65454cd7cee6b48700da0cc3bc0d61ea5e40a63064b968b6c2f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/fab298a54e356f08eec19c574ac20ca93e216825f135616bd7219fc26fff7e4b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 00:38:40.039242 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.038758    4576 manager.go:217] Machine: {Timestamp:2026-10-30 00:38:40.037878713 +0000 UTC m=+0.109202539 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099799 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d738bed7b6c55377b4e3df6637422 SystemUUID:ec2d738b-ed7b-6c55-377b-4e3df6637422 BootID:07189e76-8efe-4e7a-8f38-4e70b1aa2a7b Filesystems:[{Device:/run/containers/storage/overlay-containers/775233877f80c24f544008201c7230151e8f583c1c7df2598c9865dbd3245ec6/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22925fef216e1c43f2878aebcd1ddfc80459636cf8c28d8fc5b90b6d5793481a/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b9f3661-225a-43e3-8f77-5122261f257f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13b536f0aead184c905ca8c26683444e28599553e9300b6892080c9df3f63ae8/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a312a1ca-eba6-4a80-b070-caee6de84c04/volumes/kubernetes.io~projected/kube-api-access-tld2q DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/56a648c9-5e85-49b8-91f3-967fad2836f8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7f726d8-6d17-4370-a98c-41a979b549ae/volumes/kubernetes.io~projected/kube-api-access-69lpl DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/00bbf453-22c0-42ae-843d-7a9fafe9f7c7/volumes/kubernetes.io~projected/kube-api-access-82s67 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/87032b63-645e-4392-bd02-80b1289399c3/volumes/kubernetes.io~projected/kube-api-access-2hlvr DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1fd7b895496783c7254d47d97bb48f11a7f1e4ea4a83241f9ac89baeb2965b0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b899a78-408a-433c-8e45-c7dd7499a1f2/volumes/kubernetes.io~projected/kube-api-access-ttgzz DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1455a5657ba343086772ff987a745481368d49e09c28041761155497e7398c03/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b9f3661-225a-43e3-8f77-5122261f257f/volumes/kubernetes.io~projected/kube-api-access-49br2 DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7160b009a1359edb5160872ba3c9db507ea784b3d33c4b51daaae1b2c5a44f31/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/baa64a736c629a4a437aff098e9da447da2aa7fd991ed4a20579f1f54b737495/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/904817ce-d573-4b9e-ba53-646819696439/volumes/kubernetes.io~projected/kube-api-access-d24x6 DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/288942dd9d3b6fa02f4f63485e2327f3f3a9cf2ddfb9acfc770fff7bc2dfa4ba/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38381a16-212b-4d9c-98c2-6d81e4af3b74/volumes/kubernetes.io~projected/kube-api-access-vlfgn DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3bf88444-b628-4aba-a61e-82ae03285b9e/volumes/kubernetes.io~projected/kube-api-access-5fcdt DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b42d909c9e24bea099fed03d603ebccf4a27aa47b91d5a5161f301b7fa5d8225/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00bbf453-22c0-42ae-843d-7a9fafe9f7c7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/240f181228572cf3999d7ee380753ddfb4bc862a5d5ad6b5d7471023e9e9ca33/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e57ed1b798ee376f246a060274d760f60e8f45e005a98e45c4eb57176584691/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edfab0af-8070-4ffb-9c66-9c94399393d7/volumes/kubernetes.io~projected/kube-api-access-8zv44 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f1f3733-1c4c-492e-9cfd-443848be5859/volumes/kubernetes.io~projected/kube-api-access-4wkhb DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d10c0ce2-a04c-4f4f-9977-db48be2e5504/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40aff960-4844-46db-aa92-2efb92303076/volumes/kubernetes.io~projected/kube-api-access-d6zxt DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f7743261-4eb5-44c4-a6a0-ef3fea5e492c/volumes/kubernetes.io~projected/kube-api-access-rbvph DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2720d61591632d53e72a27378c6c0a955158c1c54bbd6aaed6f4deda9b5feaf/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a92bd22db7888c525b0fc350f89d7199dc33679ae842a52b2399800919c7dd7/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04cca4f8-e3c4-4796-bd5a-1fe9d4985408/volumes/kubernetes.io~projected/kube-api-access-j99fx DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f1f3733-1c4c-492e-9cfd-443848be5859/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff70604a5d0d291631886fdc5613ebc64bad59cf9c6989e124212e3dbe4ec061/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3fa82872f31f5f151ea52ec1d18a3e46dd820c4dd7012960311453bdba8eecf/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97b48a54-79e8-44b4-8d9e-5d1046639884/volumes/kubernetes.io~projected/kube-api-access-552tl DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f1d925dd-7561-4ee2-8e42-f38ee71296f6/volumes/kubernetes.io~projected/kube-api-access-7l6sm DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aab0c2e8f9f7615a708f0505d039b5a06854eb26daa49c0ef0ffd01191a192b0/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:22925fef216e1c4 MacAddress:7a:05:b3:a5:d0:eb Speed:10000 Mtu:8901} {Name:288942dd9d3b6fa MacAddress:0a:53:1e:1b:83:51 Speed:10000 Mtu:8901} {Name:7160b009a1359ed MacAddress:d6:94:1d:2b:e7:ec Speed:10000 Mtu:8901} {Name:9594f909b92c798 MacAddress:7e:bd:73:06:03:8d Speed:10000 Mtu:8901} {Name:9e57ed1b798ee37 MacAddress:c6:87:e0:8b:1f:e5 Speed:10000 Mtu:8901} {Name:baa64a736c629a4 MacAddress:d6:1f:ee:82:81:8f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:74:1d:59 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:89:3f:57:a1:4a Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:74:1d:59 Speed:0 Mtu:9001} {Name:ff70604a5d0d291 MacAddress:76:16:29:78:da:68 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6e:74:6b:75:f5:fe Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:b6:5a:7a:0e:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:38:40.039242 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.039249    4576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:38:40.039597 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.039376    4576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:38:40.039779 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.039748    4576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:38:40.039962 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.039779    4576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:38:40.040032 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.039965    4576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:38:40.040032 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.039973    4576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:38:40.040032 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.039987    4576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:38:40.040032 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040001    4576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:38:40.040197 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040109    4576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:38:40.040352 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040337    4576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:38:40.040481 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040464    4576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:38:40.040604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040511    4576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:38:40.040604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040529    4576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:38:40.040604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040542    4576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:38:40.040604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040556    4576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:38:40.040604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.040575    4576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:38:40.042303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.042285    4576 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:38:40.042303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.042302    4576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:38:40.044538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.043354    4576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:38:40.057530 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.057498    4576 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:38:40.057530 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.057516    4576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:38:40.057698 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.057659    4576 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 00:38:40.057948 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.057933    4576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:38:40.057996 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.057980    4576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:38:40.058113 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058102    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:38:40.058163 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058121    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:38:40.058163 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058132    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:38:40.058163 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058141    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:38:40.058163 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058158    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058167    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058176    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058186    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058197    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058206    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058221    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058252    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058287    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:38:40.058350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.058296    4576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:38:40.062663 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.062645    4576 server.go:1263] "Started kubelet"
Oct 30 00:38:40.062889 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.062862    4576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:38:40.063069 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.063017    4576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:38:40.063294 ip-10-0-130-5 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:38:40.064450 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.064343    4576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:38:40.068529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.066178    4576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:38:40.068529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.066556    4576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:38:40.068529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.066828    4576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:33:02 +0000 UTC" deadline="2028-06-21 18:25:44.954422329 +0000 UTC"
Oct 30 00:38:40.068529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.066854    4576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14417h47m4.887570599s"
Oct 30 00:38:40.068529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.067015    4576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:38:40.068529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.067022    4576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:38:40.068529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.067158    4576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:38:40.068529 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:40.067446    4576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-5.ec2.internal\" not found"
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068728    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf88444-b628-4aba-a61e-82ae03285b9e" volumeName="kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068777    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40aff960-4844-46db-aa92-2efb92303076" volumeName="kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068786    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="904817ce-d573-4b9e-ba53-646819696439" volumeName="kubernetes.io/projected/904817ce-d573-4b9e-ba53-646819696439-kube-api-access-d24x6" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068793    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7743261-4eb5-44c4-a6a0-ef3fea5e492c" volumeName="kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-config" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068807    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00bbf453-22c0-42ae-843d-7a9fafe9f7c7" volumeName="kubernetes.io/projected/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-kube-api-access-82s67" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068814    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04cca4f8-e3c4-4796-bd5a-1fe9d4985408" volumeName="kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-tmp" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068821    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04cca4f8-e3c4-4796-bd5a-1fe9d4985408" volumeName="kubernetes.io/projected/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-kube-api-access-j99fx" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068833    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b9f3661-225a-43e3-8f77-5122261f257f" volumeName="kubernetes.io/configmap/1b9f3661-225a-43e3-8f77-5122261f257f-config-volume" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068841    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38381a16-212b-4d9c-98c2-6d81e4af3b74" volumeName="kubernetes.io/projected/38381a16-212b-4d9c-98c2-6d81e4af3b74-kube-api-access-vlfgn" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068867    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56a648c9-5e85-49b8-91f3-967fad2836f8" volumeName="kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068881    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f1f3733-1c4c-492e-9cfd-443848be5859" volumeName="kubernetes.io/projected/6f1f3733-1c4c-492e-9cfd-443848be5859-kube-api-access-4wkhb" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068888    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87032b63-645e-4392-bd02-80b1289399c3" volumeName="kubernetes.io/projected/87032b63-645e-4392-bd02-80b1289399c3-kube-api-access-2hlvr" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068894    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87032b63-645e-4392-bd02-80b1289399c3" volumeName="kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068907    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97b48a54-79e8-44b4-8d9e-5d1046639884" volumeName="kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068916    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b899a78-408a-433c-8e45-c7dd7499a1f2" volumeName="kubernetes.io/projected/9b899a78-408a-433c-8e45-c7dd7499a1f2-kube-api-access-ttgzz" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068922    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7f726d8-6d17-4370-a98c-41a979b549ae" volumeName="kubernetes.io/projected/c7f726d8-6d17-4370-a98c-41a979b549ae-kube-api-access-69lpl" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068933    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10c0ce2-a04c-4f4f-9977-db48be2e5504" volumeName="kubernetes.io/configmap/d10c0ce2-a04c-4f4f-9977-db48be2e5504-konnectivity-ca" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068950    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edfab0af-8070-4ffb-9c66-9c94399393d7" volumeName="kubernetes.io/empty-dir/edfab0af-8070-4ffb-9c66-9c94399393d7-tmp-dir" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068959    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d925dd-7561-4ee2-8e42-f38ee71296f6" volumeName="kubernetes.io/projected/f1d925dd-7561-4ee2-8e42-f38ee71296f6-kube-api-access-7l6sm" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068975    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7743261-4eb5-44c4-a6a0-ef3fea5e492c" volumeName="kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-env-overrides" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068982    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b9f3661-225a-43e3-8f77-5122261f257f" volumeName="kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.068989    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38381a16-212b-4d9c-98c2-6d81e4af3b74" volumeName="kubernetes.io/configmap/38381a16-212b-4d9c-98c2-6d81e4af3b74-serviceca" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069003    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87032b63-645e-4392-bd02-80b1289399c3" volumeName="kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-multus-daemon-config" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069024    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97b48a54-79e8-44b4-8d9e-5d1046639884" volumeName="kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069031    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10c0ce2-a04c-4f4f-9977-db48be2e5504" volumeName="kubernetes.io/secret/d10c0ce2-a04c-4f4f-9977-db48be2e5504-agent-certs" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069049    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7743261-4eb5-44c4-a6a0-ef3fea5e492c" volumeName="kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069069    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf88444-b628-4aba-a61e-82ae03285b9e" volumeName="kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069100    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f1f3733-1c4c-492e-9cfd-443848be5859" volumeName="kubernetes.io/secret/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-key" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069126    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97b48a54-79e8-44b4-8d9e-5d1046639884" volumeName="kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069134    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7743261-4eb5-44c4-a6a0-ef3fea5e492c" volumeName="kubernetes.io/projected/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-kube-api-access-rbvph" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069141    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7743261-4eb5-44c4-a6a0-ef3fea5e492c" volumeName="kubernetes.io/secret/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069157    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00bbf453-22c0-42ae-843d-7a9fafe9f7c7" volumeName="kubernetes.io/secret/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-cert" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069171    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b9f3661-225a-43e3-8f77-5122261f257f" volumeName="kubernetes.io/projected/1b9f3661-225a-43e3-8f77-5122261f257f-kube-api-access-49br2" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069186    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f1f3733-1c4c-492e-9cfd-443848be5859" volumeName="kubernetes.io/configmap/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-cabundle" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069199    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a312a1ca-eba6-4a80-b070-caee6de84c04" volumeName="kubernetes.io/projected/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-api-access-tld2q" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069207    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97b48a54-79e8-44b4-8d9e-5d1046639884" volumeName="kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069247    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d925dd-7561-4ee2-8e42-f38ee71296f6" volumeName="kubernetes.io/configmap/f1d925dd-7561-4ee2-8e42-f38ee71296f6-iptables-alerter-script" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069272    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b9f3661-225a-43e3-8f77-5122261f257f" volumeName="kubernetes.io/empty-dir/1b9f3661-225a-43e3-8f77-5122261f257f-tmp-dir" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069284    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf88444-b628-4aba-a61e-82ae03285b9e" volumeName="kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069291    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf88444-b628-4aba-a61e-82ae03285b9e" volumeName="kubernetes.io/projected/3bf88444-b628-4aba-a61e-82ae03285b9e-kube-api-access-5fcdt" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069325    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a312a1ca-eba6-4a80-b070-caee6de84c04" volumeName="kubernetes.io/empty-dir/a312a1ca-eba6-4a80-b070-caee6de84c04-data-volume" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069353    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edfab0af-8070-4ffb-9c66-9c94399393d7" volumeName="kubernetes.io/projected/edfab0af-8070-4ffb-9c66-9c94399393d7-kube-api-access-8zv44" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069359    4576 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04cca4f8-e3c4-4796-bd5a-1fe9d4985408" volumeName="kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-tuned" seLinuxMountContext=""
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069367    4576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:38:40.069910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.069373    4576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:38:40.072351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072279    4576 factory.go:153] Registering CRI-O factory
Oct 30 00:38:40.072351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072297    4576 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:38:40.072753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072361    4576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:38:40.072753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072370    4576 factory.go:55] Registering systemd factory
Oct 30 00:38:40.072753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072383    4576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:38:40.072753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072447    4576 factory.go:103] Registering Raw factory
Oct 30 00:38:40.072753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072460    4576 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:38:40.072956 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072939    4576 manager.go:319] Starting recovery of all containers
Oct 30 00:38:40.073010 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.072938    4576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:38:40.073448 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.073391    4576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:38:40.084181 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.084143    4576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:38:40.100432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.100309    4576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:38:40.100432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.100335    4576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:38:40.100432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.100352    4576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:38:40.100432 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:40.100403    4576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:38:40.113750 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.113730    4576 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 00:38:40.113895 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.113838    4576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:40.114075 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.114059    4576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:40.126351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.126316    4576 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="26fe6c3a0fec07d013f33134490870d0965768e58d511d396ab8b2b715fe2cc4" exitCode=0
Oct 30 00:38:40.126351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.126354    4576 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="b0ed4bb00df473b7e086284ef6f7aeeead78bbe6f918b5a2ac18d16c974653a0" exitCode=0
Oct 30 00:38:40.126515 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.126364    4576 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="078ff424f5be733e825c12a1b080e4f9c1a1b98cdbb5fdb973057591b386f01d" exitCode=0
Oct 30 00:38:40.126515 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.126372    4576 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="3440ce8f72042d72436921cd5bd7779d645e02bf5d27785a553e198f3fe992b4" exitCode=0
Oct 30 00:38:40.126515 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.126381    4576 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="38ac1e2c28fac97eb69912552583f2b54cfec45f15bc071affd21141eaafd810" exitCode=0
Oct 30 00:38:40.126515 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.126389    4576 generic.go:356] "Generic (PLEG): container finished" podID="3bf88444-b628-4aba-a61e-82ae03285b9e" containerID="e6d226d9f7bf30fd1a078d1c8f7311b70721a42616b5e46b26c42a9a1bb5fc56" exitCode=0
Oct 30 00:38:40.129109 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.129078    4576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:40.129579 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.129559    4576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:38:40.167149 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.167106    4576 generic.go:356] "Generic (PLEG): container finished" podID="a2c5cafbcc82820178483fe6020ab6f7" containerID="b8f81471a9b31c7da9b69dd149585eb16c66f99659320a1dbd51d3e9fc1ed7c9" exitCode=0
Oct 30 00:38:40.200483 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:40.200456    4576 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 00:38:40.205177 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.205154    4576 manager.go:324] Recovery completed
Oct 30 00:38:40.220995 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.220973    4576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:38:40.220995 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.220989    4576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:38:40.221178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.221005    4576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:38:40.221178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.221126    4576 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 00:38:40.221178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.221137    4576 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 00:38:40.221178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.221155    4576 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 00:38:40.221178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.221160    4576 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 00:38:40.221178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.221165    4576 policy_none.go:50] "Start"
Oct 30 00:38:40.221178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.221174    4576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:38:40.221178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.221182    4576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:38:40.223210 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.223181    4576 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 00:38:40.223348 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.223214    4576 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 00:38:40.223348 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.223224    4576 policy_none.go:44] "Start"
Oct 30 00:38:40.228291 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.227839    4576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:38:40.233681 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.233664    4576 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:38:40.233788 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:40.233690    4576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:38:40.233788 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.233699    4576 server.go:87] "Starting device plugin registration server"
Oct 30 00:38:40.234002 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.233986    4576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:38:40.234054 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.234001    4576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:38:40.234119 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.234103    4576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:38:40.234247 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.234206    4576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:38:40.234330 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.234265    4576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:38:40.234330 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.234272    4576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:38:40.234408 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.234328    4576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:38:40.234247298Z","UUID":"db06b43a-75c6-4688-b9d1-9ff58d0e4567","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:38:40.235633 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:40.235610    4576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:38:40.235766 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.235732    4576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:38:40.235766 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.235763    4576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:38:40.334311 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.334283    4576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:38:40.338167 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.338135    4576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:38:40.338415 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.338402    4576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:38:40.338450 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.338421    4576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-5.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:38:40.338503 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.338454    4576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.378306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.378226    4576 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.378411 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.378311    4576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.400937 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.400896    4576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"]
Oct 30 00:38:40.401433 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.401402    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal" event={"ID":"f3b3e87c8faf211497319716dcdc8a4c","Type":"ContainerStarted","Data":"dcccc59a9cb5af431cdb1eef6e073cdfa73b8ab6bcae91379edb1ad96c807486"}
Oct 30 00:38:40.401497 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.401442    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal" event={"ID":"f3b3e87c8faf211497319716dcdc8a4c","Type":"ContainerStarted","Data":"5a92bd22db7888c525b0fc350f89d7199dc33679ae842a52b2399800919c7dd7"}
Oct 30 00:38:40.401497 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.401463    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal" event={"ID":"a2c5cafbcc82820178483fe6020ab6f7","Type":"ContainerStarted","Data":"ac6899c82ad5f0d671cbd9f96dc5a62f8bd85a30299396b862291e6992ffb99c"}
Oct 30 00:38:40.401497 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.401477    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal" event={"ID":"a2c5cafbcc82820178483fe6020ab6f7","Type":"ContainerDied","Data":"b8f81471a9b31c7da9b69dd149585eb16c66f99659320a1dbd51d3e9fc1ed7c9"}
Oct 30 00:38:40.401497 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.401489    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal" event={"ID":"a2c5cafbcc82820178483fe6020ab6f7","Type":"ContainerStarted","Data":"240f181228572cf3999d7ee380753ddfb4bc862a5d5ad6b5d7471023e9e9ca33"}
Oct 30 00:38:40.401691 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.401511    4576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.401691 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.401676    4576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.425927 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.425898    4576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:38:40.426059 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:40.425953    4576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-5.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.426594 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.426579    4576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:38:40.426635 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:40.426616    4576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.517136 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.517104    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3b3e87c8faf211497319716dcdc8a4c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-5.ec2.internal\" (UID: \"f3b3e87c8faf211497319716dcdc8a4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.517293 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.517158    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.517293 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.517178    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.617961 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.617924    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.618258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.617972    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.618258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.617993    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3b3e87c8faf211497319716dcdc8a4c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-5.ec2.internal\" (UID: \"f3b3e87c8faf211497319716dcdc8a4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.618258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.617995    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.618258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.618037    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f3b3e87c8faf211497319716dcdc8a4c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-5.ec2.internal\" (UID: \"f3b3e87c8faf211497319716dcdc8a4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:40.618258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:40.618079    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2c5cafbcc82820178483fe6020ab6f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" (UID: \"a2c5cafbcc82820178483fe6020ab6f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:41.041971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.041935    4576 apiserver.go:52] "Watching apiserver"
Oct 30 00:38:41.074311 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.074284    4576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:38:41.074761 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.074729    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4h2rk","openshift-multus/multus-q5pp9","openshift-cluster-node-tuning-operator/tuned-6sjgl","openshift-dns/node-resolver-6j4cc","openshift-image-registry/node-ca-frssv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs","openshift-ingress-canary/ingress-canary-mnj2x","openshift-insights/insights-runtime-extractor-7tqzf","openshift-service-ca/service-ca-5fffc97bdf-jg7w4","kube-system/konnectivity-agent-62frw","openshift-dns/dns-default-k54rc","openshift-multus/network-metrics-daemon-qp4wt","openshift-network-operator/iptables-alerter-6hsqt","openshift-ovn-kubernetes/ovnkube-node-5klc9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d","openshift-network-diagnostics/network-check-target-tds5t","kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal","openshift-multus/multus-additional-cni-plugins-l4gr4","openshift-insights/periodic-gathering-9wq8z-zs5sd"]
Oct 30 00:38:41.078372 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.078353    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:41.081734 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.081706    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.092243 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092209    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.092576 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092560    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.092677 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092664    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.092759 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092733    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.092850 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092837    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5s9q7\""
Oct 30 00:38:41.092923 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092880    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:38:41.092923 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092909    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.092985 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092922    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t5wsq\""
Oct 30 00:38:41.092985 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092925    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:38:41.092985 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.092979    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.103881 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.103857    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:38:41.103881 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.103872    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.104020 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.103916    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.105783 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.105769    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:38:41.107673 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.107659    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.107673 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.107667    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.107787 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.107661    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.107787 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.107759    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.107859 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.107849    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.107912 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.107896    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.107964 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.107922    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.108157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108136    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:38:41.108333 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108312    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.108437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108355    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:38:41.108437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108360    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.108437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108411    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:38:41.108437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108428    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:38:41.108604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108439    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.108604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108439    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:38:41.108604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108456    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.108604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108458    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.108604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108458    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:38:41.108604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108458    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:38:41.108604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108429    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.108952 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108933    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:38:41.108952 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108932    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 00:38:41.109068 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108951    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.109068 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108968    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 00:38:41.109068 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108973    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:38:41.109068 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108979    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:38:41.109068 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.108990    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:38:41.122402 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.122376    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 00:38:41.147199 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147157    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9xms5\""
Oct 30 00:38:41.147438 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147416    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:38:41.147540 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147429    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7gr5j\""
Oct 30 00:38:41.147540 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147518    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-44cn9\""
Oct 30 00:38:41.147741 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147718    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2jw8t\""
Oct 30 00:38:41.147843 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147809    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:38:41.147843 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147820    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tmln4\""
Oct 30 00:38:41.147843 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147830    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqzrd\""
Oct 30 00:38:41.147958 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147885    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-545pj\""
Oct 30 00:38:41.147958 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147885    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:38:41.147958 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147926    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kg5gj\""
Oct 30 00:38:41.147958 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147932    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 00:38:41.148129 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.147972    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-98rrr\""
Oct 30 00:38:41.148129 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148095    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2bp5j\""
Oct 30 00:38:41.148129 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148100    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ckcc7\""
Oct 30 00:38:41.148129 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148130    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q2s6t\""
Oct 30 00:38:41.148387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148134    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jxbz5\""
Oct 30 00:38:41.148387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148163    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-smcj2\""
Oct 30 00:38:41.148387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148195    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:38:41.148387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148204    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k8zgn\""
Oct 30 00:38:41.148387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148257    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:38:41.148387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148276    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9lsw5\""
Oct 30 00:38:41.148387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148330    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:38:41.148387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.148366    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:38:41.168372 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.168344    4576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:38:41.195147 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.195125    4576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:41.195300 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.195246    4576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:41.219439 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.219419    4576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:38:41.219547 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.219462    4576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-5.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:41.220021 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.220008    4576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:38:41.220068 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.220041    4576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-5.ec2.internal"
Oct 30 00:38:41.222274 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222252    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.222343 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222281    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-systemd\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.222343 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222310    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-lib-modules\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.222410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222338    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:41.222410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222388    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-node-log\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.222484 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222461    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:41.222523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222488    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-socket-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.222523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222508    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcdt\" (UniqueName: \"kubernetes.io/projected/3bf88444-b628-4aba-a61e-82ae03285b9e-kube-api-access-5fcdt\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.222579 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222525    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-bin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.222579 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222564    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tld2q\" (UniqueName: \"kubernetes.io/projected/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-api-access-tld2q\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.222710 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.222686    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56a648c9-5e85-49b8-91f3-967fad2836f8-original-pull-secret\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:41.223128 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223102    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-slash\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.223251 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223129    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:41.223251 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223129    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:41.223251 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223186    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.223251 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223217    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-system-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.223432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223265    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-multus-daemon-config\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.223432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223283    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-multus-certs\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.223432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223300    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-kubelet\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.223432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223323    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-log-socket\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.223432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223348    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:41.223432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223398    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-tmp\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223435    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223488    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223481    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-kubelet\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223525    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-tmp\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223557    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-kubelet-config\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223558    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-binary-copy\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223576    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-multus-daemon-config\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223601    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-systemd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223641    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.223680 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223665    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38381a16-212b-4d9c-98c2-6d81e4af3b74-serviceca\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223684    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223688    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-kubernetes\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223708    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-cni-binary-copy\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223723    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-socket-dir-parent\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223738    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-netns\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223754    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-cabundle\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223770    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-systemd-units\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223797    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552tl\" (UniqueName: \"kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223811    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d925dd-7561-4ee2-8e42-f38ee71296f6-host-slash\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223824    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-sys\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223839    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-cnibin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223852    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-conf-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223869    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223884    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-config\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223890    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87032b63-645e-4392-bd02-80b1289399c3-cni-binary-copy\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223899    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfgn\" (UniqueName: \"kubernetes.io/projected/38381a16-212b-4d9c-98c2-6d81e4af3b74-kube-api-access-vlfgn\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223936    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9f3661-225a-43e3-8f77-5122261f257f-tmp-dir\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223960    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a312a1ca-eba6-4a80-b070-caee6de84c04-crio-socket\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.223986    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224011    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-bin\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224042    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-cabundle\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224041    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovn-node-metrics-cert\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224068    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224086    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-device-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224099    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysconfig\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224115    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkhb\" (UniqueName: \"kubernetes.io/projected/6f1f3733-1c4c-492e-9cfd-443848be5859-kube-api-access-4wkhb\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224131    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-script-lib\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224145    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-sys-fs\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224159    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-var-lib-kubelet\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224172    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-host\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224187    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-system-cni-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224202    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-cnibin\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224218    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224255    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224282    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfab0af-8070-4ffb-9c66-9c94399393d7-tmp-dir\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224314    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-hostroot\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224386    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/38381a16-212b-4d9c-98c2-6d81e4af3b74-serviceca\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224401    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfab0af-8070-4ffb-9c66-9c94399393d7-tmp-dir\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224434    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovn-node-metrics-cert\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224496    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-config\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224496    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bf88444-b628-4aba-a61e-82ae03285b9e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224515    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f3661-225a-43e3-8f77-5122261f257f-config-volume\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224545    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-ovn\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224551    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9f3661-225a-43e3-8f77-5122261f257f-tmp-dir\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224591    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-ovnkube-script-lib\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224612    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-env-overrides\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224697    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f3661-225a-43e3-8f77-5122261f257f-config-volume\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:41.224732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224734    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lpl\" (UniqueName: \"kubernetes.io/projected/c7f726d8-6d17-4370-a98c-41a979b549ae-kube-api-access-69lpl\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224757    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-env-overrides\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224764    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-conf\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224791    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-k8s-cni-cncf-io\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224884    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-var-lib-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224917    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224955    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvph\" (UniqueName: \"kubernetes.io/projected/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-kube-api-access-rbvph\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224978    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6sm\" (UniqueName: \"kubernetes.io/projected/f1d925dd-7561-4ee2-8e42-f38ee71296f6-kube-api-access-7l6sm\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.224997    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225023    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-key\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225046    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1d925dd-7561-4ee2-8e42-f38ee71296f6-iptables-alerter-script\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225068    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38381a16-212b-4d9c-98c2-6d81e4af3b74-host\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225094    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d24x6\" (UniqueName: \"kubernetes.io/projected/904817ce-d573-4b9e-ba53-646819696439-kube-api-access-d24x6\") pod \"migrator-66df5d496b-sxwqs\" (UID: \"904817ce-d573-4b9e-ba53-646819696439\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225151    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-run\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225190    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99fx\" (UniqueName: \"kubernetes.io/projected/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-kube-api-access-j99fx\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225221    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225297    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a312a1ca-eba6-4a80-b070-caee6de84c04-data-volume\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225298    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6f1f3733-1c4c-492e-9cfd-443848be5859-signing-key\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225330    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1d925dd-7561-4ee2-8e42-f38ee71296f6-iptables-alerter-script\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225367    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a312a1ca-eba6-4a80-b070-caee6de84c04-data-volume\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225390    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d10c0ce2-a04c-4f4f-9977-db48be2e5504-konnectivity-ca\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225424    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-modprobe-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225448    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-etc-kubernetes\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225474    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225506    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225589    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225612    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-netd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225614    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d10c0ce2-a04c-4f4f-9977-db48be2e5504-konnectivity-ca\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225653    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-registration-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.225686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225685    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgzz\" (UniqueName: \"kubernetes.io/projected/9b899a78-408a-433c-8e45-c7dd7499a1f2-kube-api-access-ttgzz\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225711    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-tuned\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225721    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b9f3661-225a-43e3-8f77-5122261f257f-metrics-tls\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225798    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-multus\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225843    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-tuned\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225857    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49br2\" (UniqueName: \"kubernetes.io/projected/1b9f3661-225a-43e3-8f77-5122261f257f-kube-api-access-49br2\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225886    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.225978    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-etc-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226008    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-cert\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226046    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d10c0ce2-a04c-4f4f-9977-db48be2e5504-agent-certs\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226114    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226147    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlvr\" (UniqueName: \"kubernetes.io/projected/87032b63-645e-4392-bd02-80b1289399c3-kube-api-access-2hlvr\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226179    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-dbus\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226208    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-netns\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226218    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-cert\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226293    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfab0af-8070-4ffb-9c66-9c94399393d7-hosts-file\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226299    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d10c0ce2-a04c-4f4f-9977-db48be2e5504-agent-certs\") pod \"konnectivity-agent-62frw\" (UID: \"d10c0ce2-a04c-4f4f-9977-db48be2e5504\") " pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226318    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zv44\" (UniqueName: \"kubernetes.io/projected/edfab0af-8070-4ffb-9c66-9c94399393d7-kube-api-access-8zv44\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:41.226668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.226419    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82s67\" (UniqueName: \"kubernetes.io/projected/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-kube-api-access-82s67\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:41.242987 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.242963    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tld2q\" (UniqueName: \"kubernetes.io/projected/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-api-access-tld2q\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.242987 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.242981    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcdt\" (UniqueName: \"kubernetes.io/projected/3bf88444-b628-4aba-a61e-82ae03285b9e-kube-api-access-5fcdt\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.247051 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247023    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lpl\" (UniqueName: \"kubernetes.io/projected/c7f726d8-6d17-4370-a98c-41a979b549ae-kube-api-access-69lpl\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:41.247219 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247192    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvph\" (UniqueName: \"kubernetes.io/projected/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-kube-api-access-rbvph\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.247311 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247170    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99fx\" (UniqueName: \"kubernetes.io/projected/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-kube-api-access-j99fx\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.247311 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247068    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfgn\" (UniqueName: \"kubernetes.io/projected/38381a16-212b-4d9c-98c2-6d81e4af3b74-kube-api-access-vlfgn\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:41.247402 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247309    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552tl\" (UniqueName: \"kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl\") pod \"periodic-gathering-9wq8z-zs5sd\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") " pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:38:41.247402 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247349    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkhb\" (UniqueName: \"kubernetes.io/projected/6f1f3733-1c4c-492e-9cfd-443848be5859-kube-api-access-4wkhb\") pod \"service-ca-5fffc97bdf-jg7w4\" (UID: \"6f1f3733-1c4c-492e-9cfd-443848be5859\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jg7w4"
Oct 30 00:38:41.247510 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247497    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6sm\" (UniqueName: \"kubernetes.io/projected/f1d925dd-7561-4ee2-8e42-f38ee71296f6-kube-api-access-7l6sm\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:41.247563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247508    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zxt\" (UniqueName: \"kubernetes.io/projected/40aff960-4844-46db-aa92-2efb92303076-kube-api-access-d6zxt\") pod \"network-check-target-tds5t\" (UID: \"40aff960-4844-46db-aa92-2efb92303076\") " pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:41.247813 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247771    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49br2\" (UniqueName: \"kubernetes.io/projected/1b9f3661-225a-43e3-8f77-5122261f257f-kube-api-access-49br2\") pod \"dns-default-k54rc\" (UID: \"1b9f3661-225a-43e3-8f77-5122261f257f\") " pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:41.247933 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247906    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82s67\" (UniqueName: \"kubernetes.io/projected/00bbf453-22c0-42ae-843d-7a9fafe9f7c7-kube-api-access-82s67\") pod \"ingress-canary-mnj2x\" (UID: \"00bbf453-22c0-42ae-843d-7a9fafe9f7c7\") " pod="openshift-ingress-canary/ingress-canary-mnj2x"
Oct 30 00:38:41.247993 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247955    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlvr\" (UniqueName: \"kubernetes.io/projected/87032b63-645e-4392-bd02-80b1289399c3-kube-api-access-2hlvr\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.248040 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247995    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zv44\" (UniqueName: \"kubernetes.io/projected/edfab0af-8070-4ffb-9c66-9c94399393d7-kube-api-access-8zv44\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:41.248040 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.248018    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d24x6\" (UniqueName: \"kubernetes.io/projected/904817ce-d573-4b9e-ba53-646819696439-kube-api-access-d24x6\") pod \"migrator-66df5d496b-sxwqs\" (UID: \"904817ce-d573-4b9e-ba53-646819696439\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sxwqs"
Oct 30 00:38:41.248040 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.247997    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgzz\" (UniqueName: \"kubernetes.io/projected/9b899a78-408a-433c-8e45-c7dd7499a1f2-kube-api-access-ttgzz\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.327746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327661    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-conf\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.327746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327699    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-k8s-cni-cncf-io\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.327746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327721    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-var-lib-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327745    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327786    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-k8s-cni-cncf-io\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327797    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327797    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-var-lib-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327818    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327873    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-conf\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327886    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327941    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38381a16-212b-4d9c-98c2-6d81e4af3b74-host\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327971    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-run\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.327987    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.328005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328003    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-modprobe-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328020    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-etc-kubernetes\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328040    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-run\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328046    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328000    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38381a16-212b-4d9c-98c2-6d81e4af3b74-host\") pod \"node-ca-frssv\" (UID: \"38381a16-212b-4d9c-98c2-6d81e4af3b74\") " pod="openshift-image-registry/node-ca-frssv"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328082    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-etc-kubernetes\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328088    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328090    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328121    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.328109    4576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328136    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-modprobe-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328155    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-netd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328131    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-netd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.328207    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:41.828190839 +0000 UTC m=+1.899514683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : configmap "kube-rbac-proxy" not found
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328265    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-registration-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328292    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-multus\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328316    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328354    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-registration-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328365    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-multus\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328394    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-etc-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.328415    4576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:41.328428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328417    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.328463    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:41.828448768 +0000 UTC m=+1.899772599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328467    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-etc-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328485    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-dbus\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328501    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-netns\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328462    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328524    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfab0af-8070-4ffb-9c66-9c94399393d7-hosts-file\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328550    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328568    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-dbus\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328573    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-systemd\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328582    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfab0af-8070-4ffb-9c66-9c94399393d7-hosts-file\") pod \"node-resolver-6j4cc\" (UID: \"edfab0af-8070-4ffb-9c66-9c94399393d7\") " pod="openshift-dns/node-resolver-6j4cc"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328551    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-run-netns\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328605    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-lib-modules\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328621    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-systemd\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328631    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-node-log\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328653    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-socket-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328675    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-bin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328674    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysctl-d\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328690    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-node-log\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328697    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-slash\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328709    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-cni-bin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328713    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-lib-modules\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328728    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-system-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328734    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-socket-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328749    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-multus-certs\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328765    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-kubelet\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328733    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-slash\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328781    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-multus-certs\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328786    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-system-cni-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328799    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-kubelet\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328821    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-log-socket\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328842    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-kubelet\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328860    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-kubelet-config\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328874    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-systemd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328885    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-var-lib-kubelet\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328889    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-kubernetes\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328914    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-kubernetes\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328926    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-socket-dir-parent\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328940    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-systemd\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328872    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-log-socket\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328954    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-netns\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328956    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56a648c9-5e85-49b8-91f3-967fad2836f8-kubelet-config\") pod \"global-pull-secret-syncer-4h2rk\" (UID: \"56a648c9-5e85-49b8-91f3-967fad2836f8\") " pod="kube-system/global-pull-secret-syncer-4h2rk"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328978    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-socket-dir-parent\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328979    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-systemd-units\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.328995    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-host-run-netns\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329016    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-systemd-units\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329021    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d925dd-7561-4ee2-8e42-f38ee71296f6-host-slash\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329044    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1d925dd-7561-4ee2-8e42-f38ee71296f6-host-slash\") pod \"iptables-alerter-6hsqt\" (UID: \"f1d925dd-7561-4ee2-8e42-f38ee71296f6\") " pod="openshift-network-operator/iptables-alerter-6hsqt"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329077    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-sys\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329102    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-cnibin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329124    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-conf-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329136    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-sys\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329172    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-cnibin\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329172    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-multus-conf-dir\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329200    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a312a1ca-eba6-4a80-b070-caee6de84c04-crio-socket\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329222    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329251    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-bin\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329270    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329284    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-openvswitch\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.329263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329294    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-device-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329307    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-host-cni-bin\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329319    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a312a1ca-eba6-4a80-b070-caee6de84c04-crio-socket\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329319    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysconfig\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329339    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329349    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-sys-fs\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329364    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-device-dir\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329366    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-etc-sysconfig\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329384    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-var-lib-kubelet\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329409    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-host\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329385    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9b899a78-408a-433c-8e45-c7dd7499a1f2-sys-fs\") pod \"aws-ebs-csi-driver-node-kqs7d\" (UID: \"9b899a78-408a-433c-8e45-c7dd7499a1f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kqs7d"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329439    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-system-cni-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329470    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-system-cni-dir\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329480    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-host\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329483    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cca4f8-e3c4-4796-bd5a-1fe9d4985408-var-lib-kubelet\") pod \"tuned-6sjgl\" (UID: \"04cca4f8-e3c4-4796-bd5a-1fe9d4985408\") " pod="openshift-cluster-node-tuning-operator/tuned-6sjgl"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329487    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-cnibin\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329509    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bf88444-b628-4aba-a61e-82ae03285b9e-cnibin\") pod \"multus-additional-cni-plugins-l4gr4\" (UID: \"3bf88444-b628-4aba-a61e-82ae03285b9e\") " pod="openshift-multus/multus-additional-cni-plugins-l4gr4"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329517    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329535    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-hostroot\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329550    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-ovn\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329567    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87032b63-645e-4392-bd02-80b1289399c3-hostroot\") pod \"multus-q5pp9\" (UID: \"87032b63-645e-4392-bd02-80b1289399c3\") " pod="openshift-multus/multus-q5pp9"
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.329588    4576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.329629    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:41.829620297 +0000 UTC m=+1.900944111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : secret "metrics-daemon-secret" not found
Oct 30 00:38:41.330988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.329589    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7743261-4eb5-44c4-a6a0-ef3fea5e492c-run-ovn\") pod \"ovnkube-node-5klc9\" (UID: \"f7743261-4eb5-44c4-a6a0-ef3fea5e492c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:41.381737 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.381714    4576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:38:41.440094 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.440055    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:41.833601 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.833563    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:41.833601 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.833608    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.833855 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.833629    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:41.833855 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.833706    4576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:38:41.833855 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.833731    4576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:41.833855 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.833736    4576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:38:41.833855 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.833775    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:42.833760721 +0000 UTC m=+2.905084549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : configmap "kube-rbac-proxy" not found
Oct 30 00:38:41.833855 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.833797    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:42.833781996 +0000 UTC m=+2.905105812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : secret "metrics-daemon-secret" not found
Oct 30 00:38:41.833855 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:41.833815    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:42.833806281 +0000 UTC m=+2.905130093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:41.879849 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:41.879822    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:42.200908 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:42.199837    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k54rc" event={"ID":"1b9f3661-225a-43e3-8f77-5122261f257f","Type":"ContainerStarted","Data":"8e1f500b8c0a986ae9501c4d225315e8db7192a22dcef80332c5c8bd4ade35d5"}
Oct 30 00:38:42.285784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:42.285747    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:42.498786 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:42.498743    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:42.811898 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:42.811815    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:42.840363 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:42.840327    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:42.840502 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:42.840381    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:42.840502 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:42.840413    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:42.840502 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:42.840476    4576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:38:42.840646 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:42.840512    4576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:38:42.840646 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:42.840516    4576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:42.840646 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:42.840542    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:44.840527374 +0000 UTC m=+4.911851200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : secret "metrics-daemon-secret" not found
Oct 30 00:38:42.840646 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:42.840570    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:44.840556731 +0000 UTC m=+4.911880580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : configmap "kube-rbac-proxy" not found
Oct 30 00:38:42.840646 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:42.840584    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:44.840577093 +0000 UTC m=+4.911900907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:42.931955 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:42.931919    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:43.061018 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:43.060968    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:43.204879 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:43.204845    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k54rc" event={"ID":"1b9f3661-225a-43e3-8f77-5122261f257f","Type":"ContainerStarted","Data":"91a45be1d92eb564d6815263cd6c80183074b39ade4609e58acefd57281241cb"}
Oct 30 00:38:43.205259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:43.204951    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:43.646039 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:43.645960    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:43.782344 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:43.782307    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:44.387440 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:44.387385    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:44.556338 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:44.556300    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:44.758043 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:44.758006    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:44.855122 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:44.855088    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:44.855317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:44.855131    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:44.855317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:44.855152    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:44.855317 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:44.855262    4576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:44.855450 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:44.855336    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls podName:a312a1ca-eba6-4a80-b070-caee6de84c04 nodeName:}" failed. No retries permitted until 2026-10-30 00:38:48.85532384 +0000 UTC m=+8.926647666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7tqzf" (UID: "a312a1ca-eba6-4a80-b070-caee6de84c04") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:38:44.855450 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:44.855265    4576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:38:44.855450 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:38:44.855421    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs podName:c7f726d8-6d17-4370-a98c-41a979b549ae nodeName:}" failed. No retries permitted until 2026-10-30 00:38:48.855403986 +0000 UTC m=+8.926727802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs") pod "network-metrics-daemon-qp4wt" (UID: "c7f726d8-6d17-4370-a98c-41a979b549ae") : secret "metrics-daemon-secret" not found
Oct 30 00:38:44.855696 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:44.855680    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a312a1ca-eba6-4a80-b070-caee6de84c04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:44.920464 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:44.920430    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:45.054203 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:45.054115    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:45.649109 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:45.649055    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:46.239547 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:46.239515    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:46.426647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:46.426621    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:46.828106 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:46.828077    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:46.930202 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:46.930179    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:46.930446 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:46.930433    4576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:38:47.029029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:47.028988    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:47.431360 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:47.431308    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:47.649645 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:47.649612    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:48.881877 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:48.881840    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:48.882280 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:48.881890    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:48.882280 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:48.882258    4576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:38:48.884396 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:48.884375    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a312a1ca-eba6-4a80-b070-caee6de84c04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tqzf\" (UID: \"a312a1ca-eba6-4a80-b070-caee6de84c04\") " pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:48.884525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:48.884505    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7f726d8-6d17-4370-a98c-41a979b549ae-metrics-certs\") pod \"network-metrics-daemon-qp4wt\" (UID: \"c7f726d8-6d17-4370-a98c-41a979b549ae\") " pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:49.178732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:49.178651    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qp4wt"
Oct 30 00:38:49.184657 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:49.184509    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7tqzf"
Oct 30 00:38:49.480014 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:49.479980    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7tqzf"]
Oct 30 00:38:49.483342 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:49.483314    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda312a1ca_eba6_4a80_b070_caee6de84c04.slice/crio-006f5c466729692e273593a080ed921932273f86c5b751c4441ab3b10280e842 WatchSource:0}: Error finding container 006f5c466729692e273593a080ed921932273f86c5b751c4441ab3b10280e842: Status 404 returned error can't find the container with id 006f5c466729692e273593a080ed921932273f86c5b751c4441ab3b10280e842
Oct 30 00:38:49.493789 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:49.493769    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qp4wt"]
Oct 30 00:38:49.495385 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:49.495358    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f726d8_6d17_4370_a98c_41a979b549ae.slice/crio-8d731b252ceacb45cabdeea70d0e4eb2b0036ac03232431e6884f089c874f7ba WatchSource:0}: Error finding container 8d731b252ceacb45cabdeea70d0e4eb2b0036ac03232431e6884f089c874f7ba: Status 404 returned error can't find the container with id 8d731b252ceacb45cabdeea70d0e4eb2b0036ac03232431e6884f089c874f7ba
Oct 30 00:38:49.727167 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:49.727132    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:49.960871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:49.960824    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:50.096854 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.096826    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:38:50.107923 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.107884    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:50.228368 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.228280    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qp4wt" event={"ID":"c7f726d8-6d17-4370-a98c-41a979b549ae","Type":"ContainerStarted","Data":"8d731b252ceacb45cabdeea70d0e4eb2b0036ac03232431e6884f089c874f7ba"}
Oct 30 00:38:50.230433 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.230404    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tqzf" event={"ID":"a312a1ca-eba6-4a80-b070-caee6de84c04","Type":"ContainerStarted","Data":"4cf0883e9fff7159297af9c43aefb24d2720e42b5554a367a58fd98ba6bfd30b"}
Oct 30 00:38:50.230570 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.230465    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tqzf" event={"ID":"a312a1ca-eba6-4a80-b070-caee6de84c04","Type":"ContainerStarted","Data":"4bd534386658de30ccb06927e5190c5d37c864f861710124b17d1ea9b45b673d"}
Oct 30 00:38:50.230570 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.230493    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tqzf" event={"ID":"a312a1ca-eba6-4a80-b070-caee6de84c04","Type":"ContainerStarted","Data":"006f5c466729692e273593a080ed921932273f86c5b751c4441ab3b10280e842"}
Oct 30 00:38:50.289486 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.289136    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:50.443110 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.443075    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:50.730323 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.730293    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5klc9"
Oct 30 00:38:50.862758 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.862683    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-646d6d98c-rp7xm"]
Oct 30 00:38:50.865870 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.865847    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:50.902589 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.902556    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:50.913553 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.913517    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:38:50.913553 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.913568    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:38:50.913847 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.913604    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qbmx5\""
Oct 30 00:38:50.913847 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.913627    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:38:50.913847 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.913678    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:38:50.914297 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.914268    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646d6d98c-rp7xm"]
Oct 30 00:38:50.935053 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.935025    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-6l59b"]
Oct 30 00:38:50.938101 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.938074    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:38:50.959993 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.959971    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 00:38:50.959993 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.959971    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 00:38:50.968860 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.968839    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6l59b"]
Oct 30 00:38:50.999514 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.999489    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-registry-tls\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:50.999660 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.999527    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7c9b38a-278c-43ef-92ba-9987367dbb13-image-registry-private-configuration\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:50.999660 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.999555    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7c9b38a-278c-43ef-92ba-9987367dbb13-installation-pull-secrets\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:50.999780 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.999708    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7c9b38a-278c-43ef-92ba-9987367dbb13-registry-certificates\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:50.999780 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.999743    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c9b38a-278c-43ef-92ba-9987367dbb13-trusted-ca\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:50.999863 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.999779    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7c9b38a-278c-43ef-92ba-9987367dbb13-ca-trust-extracted\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:50.999863 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.999805    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-bound-sa-token\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:50.999863 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:50.999845    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fsw\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-kube-api-access-29fsw\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.097061 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.097006    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:51.100959 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.100935    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7c9b38a-278c-43ef-92ba-9987367dbb13-ca-trust-extracted\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.101096 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.100974    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ce63-11e7-447b-a015-eda0fa7adeab-tmp\") pod \"downloads-6dd789c8b7-6l59b\" (UID: \"a3e0ce63-11e7-447b-a015-eda0fa7adeab\") " pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:38:51.101096 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.100994    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-registry-tls\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.101096 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.101022    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7c9b38a-278c-43ef-92ba-9987367dbb13-image-registry-private-configuration\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.101096 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.101089    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcxt\" (UniqueName: \"kubernetes.io/projected/a3e0ce63-11e7-447b-a015-eda0fa7adeab-kube-api-access-nqcxt\") pod \"downloads-6dd789c8b7-6l59b\" (UID: \"a3e0ce63-11e7-447b-a015-eda0fa7adeab\") " pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:38:51.101327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.101127    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7c9b38a-278c-43ef-92ba-9987367dbb13-registry-certificates\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.101327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.101149    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c9b38a-278c-43ef-92ba-9987367dbb13-trusted-ca\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.101327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.101171    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-bound-sa-token\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.101327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.101201    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fsw\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-kube-api-access-29fsw\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.101327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.101308    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7c9b38a-278c-43ef-92ba-9987367dbb13-installation-pull-secrets\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.101626 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.101413    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7c9b38a-278c-43ef-92ba-9987367dbb13-ca-trust-extracted\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.102208 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.102182    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7c9b38a-278c-43ef-92ba-9987367dbb13-registry-certificates\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.102575 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.102538    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c9b38a-278c-43ef-92ba-9987367dbb13-trusted-ca\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.103929 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.103908    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7c9b38a-278c-43ef-92ba-9987367dbb13-image-registry-private-configuration\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.104013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.103981    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-registry-tls\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.104170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.104150    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7c9b38a-278c-43ef-92ba-9987367dbb13-installation-pull-secrets\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.126992 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.126936    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-bound-sa-token\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.127304 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.127283    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fsw\" (UniqueName: \"kubernetes.io/projected/d7c9b38a-278c-43ef-92ba-9987367dbb13-kube-api-access-29fsw\") pod \"image-registry-646d6d98c-rp7xm\" (UID: \"d7c9b38a-278c-43ef-92ba-9987367dbb13\") " pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.178825 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.178794    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:51.202244 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.202203    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ce63-11e7-447b-a015-eda0fa7adeab-tmp\") pod \"downloads-6dd789c8b7-6l59b\" (UID: \"a3e0ce63-11e7-447b-a015-eda0fa7adeab\") " pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:38:51.202373 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.202259    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcxt\" (UniqueName: \"kubernetes.io/projected/a3e0ce63-11e7-447b-a015-eda0fa7adeab-kube-api-access-nqcxt\") pod \"downloads-6dd789c8b7-6l59b\" (UID: \"a3e0ce63-11e7-447b-a015-eda0fa7adeab\") " pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:38:51.202657 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.202630    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ce63-11e7-447b-a015-eda0fa7adeab-tmp\") pod \"downloads-6dd789c8b7-6l59b\" (UID: \"a3e0ce63-11e7-447b-a015-eda0fa7adeab\") " pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:38:51.225035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.225008    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcxt\" (UniqueName: \"kubernetes.io/projected/a3e0ce63-11e7-447b-a015-eda0fa7adeab-kube-api-access-nqcxt\") pod \"downloads-6dd789c8b7-6l59b\" (UID: \"a3e0ce63-11e7-447b-a015-eda0fa7adeab\") " pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:38:51.237485 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.236998    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qp4wt" event={"ID":"c7f726d8-6d17-4370-a98c-41a979b549ae","Type":"ContainerStarted","Data":"aed8d22af480df2574b17cb5997ce15abb17a3a52ea52b484e19b48cd5c757b3"}
Oct 30 00:38:51.237485 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.237059    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qp4wt" event={"ID":"c7f726d8-6d17-4370-a98c-41a979b549ae","Type":"ContainerStarted","Data":"f5e99bb8da0e266d9df06cd68a68fc09e704f825bb8af55a0b485a61586582eb"}
Oct 30 00:38:51.250149 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.250114    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:38:51.289949 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.289917    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:51.510178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.509580    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-646d6d98c-rp7xm"]
Oct 30 00:38:51.512963 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:38:51.512930    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c9b38a_278c_43ef_92ba_9987367dbb13.slice/crio-e6c2e40979a9b6095a4c5cc8dbe8329af54aca6aa3de496b6734444c5c50c9ce WatchSource:0}: Error finding container e6c2e40979a9b6095a4c5cc8dbe8329af54aca6aa3de496b6734444c5c50c9ce: Status 404 returned error can't find the container with id e6c2e40979a9b6095a4c5cc8dbe8329af54aca6aa3de496b6734444c5c50c9ce
Oct 30 00:38:51.559648 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.559623    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6l59b"]
Oct 30 00:38:51.562581 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.562556    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:51.711984 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:51.711952    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:52.243095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:52.242524    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646d6d98c-rp7xm" event={"ID":"d7c9b38a-278c-43ef-92ba-9987367dbb13","Type":"ContainerStarted","Data":"8538a914cf83c71c64ef3cbd2788ab5043c57508af3136b651aab2b0d2eace27"}
Oct 30 00:38:52.243095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:52.242637    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:38:52.243095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:52.242665    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-646d6d98c-rp7xm" event={"ID":"d7c9b38a-278c-43ef-92ba-9987367dbb13","Type":"ContainerStarted","Data":"e6c2e40979a9b6095a4c5cc8dbe8329af54aca6aa3de496b6734444c5c50c9ce"}
Oct 30 00:38:52.245105 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:52.245072    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tqzf" event={"ID":"a312a1ca-eba6-4a80-b070-caee6de84c04","Type":"ContainerStarted","Data":"77572609afb6a717a51fdf8e9e0066b168aa336ba25739be030b74501bf0d28b"}
Oct 30 00:38:52.246533 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:52.246504    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6l59b" event={"ID":"a3e0ce63-11e7-447b-a015-eda0fa7adeab","Type":"ContainerStarted","Data":"263035560b798951a8604d68335750048786cb0b70a0c0b08787f02cdfc5759d"}
Oct 30 00:38:52.299872 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:52.299808    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-646d6d98c-rp7xm" podStartSLOduration=2.299791221 podStartE2EDuration="2.299791221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:52.299213115 +0000 UTC m=+12.370536952" watchObservedRunningTime="2026-10-30 00:38:52.299791221 +0000 UTC m=+12.371115059"
Oct 30 00:38:53.210953 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:53.210917    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k54rc"
Oct 30 00:38:54.766029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:54.765984    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:54.961722 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:54.961686    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:55.229158 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:55.229123    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:55.423071 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:55.423037    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:56.677589 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:56.677558    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-62frw"
Oct 30 00:38:58.284023 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:58.283923    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:58.478354 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:58.478317    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:58.702115 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:58.702067    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:38:59.942186 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:38:59.942133    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:00.124917 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:00.124876    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:00.993515 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:00.993479    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:02.292945 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:02.292905    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:02.429182 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:02.429140    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:02.740733 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:02.740690    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:02.904721 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:02.904546    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:03.079717 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.079631    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fllj8"]
Oct 30 00:39:03.084186 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.084157    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.103333 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.103297    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:03.120340 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.120269    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:39:03.120340 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.120297    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:39:03.120340 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.120307    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:39:03.120340 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.120296    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p7bx4\""
Oct 30 00:39:03.120340 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.120314    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:39:03.120634 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.120373    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:39:03.120804 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.120791    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:39:03.199420 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199382    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-root\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.199608 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199436    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-accelerators-collector-config\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.199608 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199489    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7rb\" (UniqueName: \"kubernetes.io/projected/ecfbfe46-06c9-488c-8016-72d31176fdd5-kube-api-access-sg7rb\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.199608 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199584    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecfbfe46-06c9-488c-8016-72d31176fdd5-metrics-client-ca\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.199714 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199618    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-wtmp\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.199714 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199648    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-tls\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.199714 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199678    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-sys\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.199806 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199759    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-textfile\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.199806 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.199795    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300366 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300333    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300381    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-root\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300414    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-accelerators-collector-config\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300435    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7rb\" (UniqueName: \"kubernetes.io/projected/ecfbfe46-06c9-488c-8016-72d31176fdd5-kube-api-access-sg7rb\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300470    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecfbfe46-06c9-488c-8016-72d31176fdd5-metrics-client-ca\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300537    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-root\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300581    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-wtmp\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300614    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-tls\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300642    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-sys\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.300866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300671    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-textfile\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.301312 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300901    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-wtmp\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.301312 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300954    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-textfile\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.301312 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.300959    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecfbfe46-06c9-488c-8016-72d31176fdd5-sys\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.301312 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.301096    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-accelerators-collector-config\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.301506 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.301486    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecfbfe46-06c9-488c-8016-72d31176fdd5-metrics-client-ca\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.303594 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.303567    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.303720 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.303668    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ecfbfe46-06c9-488c-8016-72d31176fdd5-node-exporter-tls\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.328009 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.327980    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7rb\" (UniqueName: \"kubernetes.io/projected/ecfbfe46-06c9-488c-8016-72d31176fdd5-kube-api-access-sg7rb\") pod \"node-exporter-fllj8\" (UID: \"ecfbfe46-06c9-488c-8016-72d31176fdd5\") " pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.402000 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.401905    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fllj8"
Oct 30 00:39:03.416126 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.416099    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:03.416304 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:39:03.416226    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfbfe46_06c9_488c_8016_72d31176fdd5.slice/crio-416549a35bdcb6c823320842521456c141b4712696a5752c814a6304e37d9570 WatchSource:0}: Error finding container 416549a35bdcb6c823320842521456c141b4712696a5752c814a6304e37d9570: Status 404 returned error can't find the container with id 416549a35bdcb6c823320842521456c141b4712696a5752c814a6304e37d9570
Oct 30 00:39:03.579015 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.578983    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:03.707483 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:03.707442    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:04.141312 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:04.139024    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:04.287512 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:04.287473    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fllj8" event={"ID":"ecfbfe46-06c9-488c-8016-72d31176fdd5","Type":"ContainerStarted","Data":"416549a35bdcb6c823320842521456c141b4712696a5752c814a6304e37d9570"}
Oct 30 00:39:04.434822 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:04.434746    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:04.585997 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:04.585950    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:04.844395 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:04.844362    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:04.977117 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:04.977037    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:05.292804 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:05.292711    4576 generic.go:356] "Generic (PLEG): container finished" podID="ecfbfe46-06c9-488c-8016-72d31176fdd5" containerID="8deebabce29ec98c357df073fdea9bfb09de195704153f1a36041c68b634c003" exitCode=0
Oct 30 00:39:05.292804 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:05.292788    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fllj8" event={"ID":"ecfbfe46-06c9-488c-8016-72d31176fdd5","Type":"ContainerDied","Data":"8deebabce29ec98c357df073fdea9bfb09de195704153f1a36041c68b634c003"}
Oct 30 00:39:06.242886 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:06.242851    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:06.300447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:06.300413    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fllj8" event={"ID":"ecfbfe46-06c9-488c-8016-72d31176fdd5","Type":"ContainerStarted","Data":"5cd0597eeee57654a9444fd0ac8216ede598e5e41cd1a9993da31d45960a8c72"}
Oct 30 00:39:06.300614 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:06.300464    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fllj8" event={"ID":"ecfbfe46-06c9-488c-8016-72d31176fdd5","Type":"ContainerStarted","Data":"012df1957a393f7c91acda053dbeecf8b5485e05ccea1d5f0f522ac0b4024a7e"}
Oct 30 00:39:06.509964 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:06.509883    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:06.655128 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:06.655089    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:06.886060 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:06.885964    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:07.808435 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.808392    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fllj8" podStartSLOduration=3.51752493 podStartE2EDuration="4.808378781s" totalImagesPullingTime="1.199704774s" totalInitContainerRuntime="91.149077ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:06.397518002 +0000 UTC m=+26.468841837" watchObservedRunningTime="2026-10-30 00:39:07.808378781 +0000 UTC m=+27.879702616"
Oct 30 00:39:07.808760 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.808540    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"]
Oct 30 00:39:07.813730 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.813705    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:07.829810 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.829788    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:07.830594 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.830573    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"]
Oct 30 00:39:07.860469 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.860448    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 00:39:07.860469 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.860448    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pfg7d\""
Oct 30 00:39:07.929976 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.929937    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:07.935909 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.935887    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjdw\" (UniqueName: \"kubernetes.io/projected/4219ea51-a9b2-4276-b929-ad28209dff06-kube-api-access-2zjdw\") pod \"monitoring-plugin-77f4f76594-8l77x\" (UID: \"4219ea51-a9b2-4276-b929-ad28209dff06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:07.936034 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:07.935921    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4219ea51-a9b2-4276-b929-ad28209dff06-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8l77x\" (UID: \"4219ea51-a9b2-4276-b929-ad28209dff06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:08.036445 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.036369    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjdw\" (UniqueName: \"kubernetes.io/projected/4219ea51-a9b2-4276-b929-ad28209dff06-kube-api-access-2zjdw\") pod \"monitoring-plugin-77f4f76594-8l77x\" (UID: \"4219ea51-a9b2-4276-b929-ad28209dff06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:08.036445 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.036415    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4219ea51-a9b2-4276-b929-ad28209dff06-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8l77x\" (UID: \"4219ea51-a9b2-4276-b929-ad28209dff06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:08.038920 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.038897    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4219ea51-a9b2-4276-b929-ad28209dff06-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-8l77x\" (UID: \"4219ea51-a9b2-4276-b929-ad28209dff06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:08.057362 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.057333    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjdw\" (UniqueName: \"kubernetes.io/projected/4219ea51-a9b2-4276-b929-ad28209dff06-kube-api-access-2zjdw\") pod \"monitoring-plugin-77f4f76594-8l77x\" (UID: \"4219ea51-a9b2-4276-b929-ad28209dff06\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:08.152029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.151973    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:08.310663 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.310121    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6l59b" event={"ID":"a3e0ce63-11e7-447b-a015-eda0fa7adeab","Type":"ContainerStarted","Data":"23f33cc7052064e446987c21c200ff0d3a6fda7be2a770c5f5da337e6b236d5e"}
Oct 30 00:39:08.310663 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.310362    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:39:08.331613 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.331581    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-6l59b"
Oct 30 00:39:08.356094 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.356042    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-6l59b" podStartSLOduration=2.182319037 podStartE2EDuration="18.356025709s" totalImagesPullingTime="16.173706672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:08.35569539 +0000 UTC m=+28.427019225" watchObservedRunningTime="2026-10-30 00:39:08.356025709 +0000 UTC m=+28.427349545"
Oct 30 00:39:08.465945 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:08.465916    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"]
Oct 30 00:39:08.474578 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:39:08.474524    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4219ea51_a9b2_4276_b929_ad28209dff06.slice/crio-2c8e74dc870b9c284374e0f913c97348044015faccd3b2fea793b03b37f77dde WatchSource:0}: Error finding container 2c8e74dc870b9c284374e0f913c97348044015faccd3b2fea793b03b37f77dde: Status 404 returned error can't find the container with id 2c8e74dc870b9c284374e0f913c97348044015faccd3b2fea793b03b37f77dde
Oct 30 00:39:09.327556 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.327520    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x" event={"ID":"4219ea51-a9b2-4276-b929-ad28209dff06","Type":"ContainerStarted","Data":"2c8e74dc870b9c284374e0f913c97348044015faccd3b2fea793b03b37f77dde"}
Oct 30 00:39:09.347717 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.347678    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:09.473998 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.473915    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:09.936602 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.936565    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:39:09.959722 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.959679    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:09.959990 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.959968    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:39:09.960152 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.960135    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:09.982455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.982324    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:39:09.982455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.982352    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:39:09.982455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.982376    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:39:09.982455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.982390    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:39:09.982455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.982418    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:39:09.982455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.982437    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:39:09.982455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.982446    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:39:09.982864 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.982479    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:39:09.995993 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.995961    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:39:09.996156 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.996033    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9qhgeickgd32l\""
Oct 30 00:39:09.996274 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.996256    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 00:39:09.996355 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.996312    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wftph\""
Oct 30 00:39:09.996441 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:09.996423    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:39:10.000344 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.000320    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:39:10.010216 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.010190    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:39:10.024899 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.024875    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:39:10.025101 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.024855    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:39:10.099526 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.099496    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tds5t"
Oct 30 00:39:10.155903 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.155868    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.155973    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156017    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156059    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156278 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156183    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-config-out\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156278 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156214    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156397 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156294    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156397 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156320    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156397 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156344    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156397 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156384    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156601 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156419    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156601 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156484    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156601 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156534    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-config\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156601 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156575    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xzh\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-kube-api-access-t2xzh\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156797 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156601    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-web-config\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156797 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156637    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156797 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156660    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156797 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156679    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156797 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156702    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.156797 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.156732    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257136 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257105    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-config\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257136 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257141    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xzh\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-kube-api-access-t2xzh\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257175    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-web-config\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257202    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257243    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257267    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257290    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257318    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257346    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257365    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257396    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257423    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257465    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-config-out\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257480    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257523    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257542    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257565    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257584    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257608    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.257777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.257631    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.260027 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.259322    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.260027 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.259322    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.260027 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.259653    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.260302 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.260031    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.260685 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.260463    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.260685 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.260659    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.261509 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.261481    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.261835 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.261815    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.262882 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.262857    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.264730 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.264622    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.266027 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.266001    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.266324 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.266303    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.269094 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.268991    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-config\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.269094 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.269068    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.269659 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.269636    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.270138 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.270116    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-config-out\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.270254 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.270124    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.272312 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.272263    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-web-config\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.273075 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.273046    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.282992 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.282965    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xzh\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-kube-api-access-t2xzh\") pod \"prometheus-k8s-0\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.441861 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.441830    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:10.804500 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:10.804325    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:39:10.809176 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:39:10.809146    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d29c6ab_8547_4cea_8828_24da7d652f22.slice/crio-afecb9f745188d00f3d4a9d56c4ae04bac75cb914144de7a6e697c90b601ae3b WatchSource:0}: Error finding container afecb9f745188d00f3d4a9d56c4ae04bac75cb914144de7a6e697c90b601ae3b: Status 404 returned error can't find the container with id afecb9f745188d00f3d4a9d56c4ae04bac75cb914144de7a6e697c90b601ae3b
Oct 30 00:39:11.344944 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:11.344415    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x" event={"ID":"4219ea51-a9b2-4276-b929-ad28209dff06","Type":"ContainerStarted","Data":"f7f1a1615d14c2caa034502cf381aaa362fdf82beae798bbc497255d8d51ed8e"}
Oct 30 00:39:11.344944 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:11.344913    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:11.346316 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:11.346211    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerStarted","Data":"afecb9f745188d00f3d4a9d56c4ae04bac75cb914144de7a6e697c90b601ae3b"}
Oct 30 00:39:11.350852 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:11.350828    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x"
Oct 30 00:39:11.388066 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:11.388015    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-8l77x" podStartSLOduration=2.528528874 podStartE2EDuration="4.387998659s" totalImagesPullingTime="1.859469785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:11.387474665 +0000 UTC m=+31.458798531" watchObservedRunningTime="2026-10-30 00:39:11.387998659 +0000 UTC m=+31.459322494"
Oct 30 00:39:13.254760 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:13.254726    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-646d6d98c-rp7xm"
Oct 30 00:39:13.355702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:13.355657    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerID="56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9" exitCode=0
Oct 30 00:39:13.355877 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:13.355737    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerDied","Data":"56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9"}
Oct 30 00:39:13.478511 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:13.478476    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:13.609479 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:13.609376    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:18.378665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:18.378632    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerStarted","Data":"f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801"}
Oct 30 00:39:18.379157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:18.378702    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerStarted","Data":"dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484"}
Oct 30 00:39:18.386979 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:39:18.386952    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:39:19.427191 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:19.427150    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:19.562762 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:19.562727    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:19.688917 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:19.688891    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:19.813093 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:19.813067    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:20.388689 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:20.388618    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerStarted","Data":"905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f"}
Oct 30 00:39:20.388689 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:20.388679    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerStarted","Data":"94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd"}
Oct 30 00:39:20.388866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:20.388704    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerStarted","Data":"645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af"}
Oct 30 00:39:20.388866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:20.388728    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerStarted","Data":"e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6"}
Oct 30 00:39:20.442414 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:20.442389    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:39:20.488690 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:20.488639    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.878556692 podStartE2EDuration="11.488625571s" totalImagesPullingTime="7.585084324s" totalInitContainerRuntime="24.984555ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:20.47417261 +0000 UTC m=+40.545496445" watchObservedRunningTime="2026-10-30 00:39:20.488625571 +0000 UTC m=+40.559949396"
Oct 30 00:39:23.179109 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:23.179032    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:23.291599 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:23.291564    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:25.598380 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:25.598343    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:25.734941 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:25.734910    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:26.280286 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:26.280251    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:26.393796 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:26.393763    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:26.890412 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:26.890379    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:27.044182 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:27.044148    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:34.687299 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:34.687259    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:34.800003 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:34.799970    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:48.379876 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:39:48.379837    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:39:49.683311 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:49.683265    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:49.799348 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:49.799312    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:52.249788 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:52.249755    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:52.361018 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:52.360990    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:39:59.516799 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:59.516765    4576 generic.go:356] "Generic (PLEG): container finished" podID="97b48a54-79e8-44b4-8d9e-5d1046639884" containerID="3c6b081cf99805a5a0ab50fd0550bf4da7e92585cd0dada73ce169e23c112ad6" exitCode=0
Oct 30 00:39:59.517334 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:39:59.516840    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd" event={"ID":"97b48a54-79e8-44b4-8d9e-5d1046639884","Type":"ContainerDied","Data":"3c6b081cf99805a5a0ab50fd0550bf4da7e92585cd0dada73ce169e23c112ad6"}
Oct 30 00:40:00.640730 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.640707    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:40:00.765416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.765388    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl\" (UniqueName: \"kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl\") pod \"97b48a54-79e8-44b4-8d9e-5d1046639884\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") "
Oct 30 00:40:00.765629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.765476    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle\") pod \"97b48a54-79e8-44b4-8d9e-5d1046639884\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") "
Oct 30 00:40:00.765629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.765505    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle\") pod \"97b48a54-79e8-44b4-8d9e-5d1046639884\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") "
Oct 30 00:40:00.765629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.765552    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path\") pod \"97b48a54-79e8-44b4-8d9e-5d1046639884\" (UID: \"97b48a54-79e8-44b4-8d9e-5d1046639884\") "
Oct 30 00:40:00.765879 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.765857    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle" pod "97b48a54-79e8-44b4-8d9e-5d1046639884" (UID: "97b48a54-79e8-44b4-8d9e-5d1046639884"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:00.765932 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.765874    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path" pod "97b48a54-79e8-44b4-8d9e-5d1046639884" (UID: "97b48a54-79e8-44b4-8d9e-5d1046639884"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:40:00.765932 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.765909    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle" pod "97b48a54-79e8-44b4-8d9e-5d1046639884" (UID: "97b48a54-79e8-44b4-8d9e-5d1046639884"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:00.767701 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.767672    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl" pod "97b48a54-79e8-44b4-8d9e-5d1046639884" (UID: "97b48a54-79e8-44b4-8d9e-5d1046639884"). InnerVolumeSpecName "kube-api-access-552tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:00.866784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.866751    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552tl\" (UniqueName: \"kubernetes.io/projected/97b48a54-79e8-44b4-8d9e-5d1046639884-kube-api-access-552tl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:00.866784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.866779    4576 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-service-ca-bundle\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:00.866784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.866788    4576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97b48a54-79e8-44b4-8d9e-5d1046639884-trusted-ca-bundle\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:00.866995 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:00.866799    4576 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/97b48a54-79e8-44b4-8d9e-5d1046639884-archives-path\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:01.524129 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:01.524101    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd"
Oct 30 00:40:01.524129 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:01.524115    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9wq8z-zs5sd" event={"ID":"97b48a54-79e8-44b4-8d9e-5d1046639884","Type":"ContainerDied","Data":"9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1"}
Oct 30 00:40:01.524542 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:01.524159    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9594f909b92c798785a16f9df738121116e785655d86a9ce0354dc0b30cae7d1"
Oct 30 00:40:02.106007 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:02.105977    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:02.248039 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:02.248000    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:03.841290 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:03.841223    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:03.859905 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:03.859881    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:04.549367 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:04.549343    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:09.698492 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:09.698453    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:18.379470 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:18.379427    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:40:19.305319 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.305287    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:40:19.305767 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.305730    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy" containerID="cri-o://94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd" gracePeriod=600
Oct 30 00:40:19.305767 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.305730    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="config-reloader" containerID="cri-o://f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801" gracePeriod=600
Oct 30 00:40:19.306026 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.305729    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="prometheus" containerID="cri-o://dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484" gracePeriod=600
Oct 30 00:40:19.306026 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.305766    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy-thanos" containerID="cri-o://905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f" gracePeriod=600
Oct 30 00:40:19.306026 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.305802    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="thanos-sidecar" containerID="cri-o://e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6" gracePeriod=600
Oct 30 00:40:19.306026 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.305751    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy-web" containerID="cri-o://645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af" gracePeriod=600
Oct 30 00:40:19.534609 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:19.534557    4576 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 00:40:19.535006 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:19.534628    4576 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 00:40:19.535006 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:19.534645    4576 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 00:40:19.535006 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.534660    4576 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Oct 30 00:40:19.588112 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588043    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerID="905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f" exitCode=0
Oct 30 00:40:19.588112 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588068    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerID="94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd" exitCode=0
Oct 30 00:40:19.588112 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588075    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerID="e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6" exitCode=0
Oct 30 00:40:19.588112 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588081    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerID="f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801" exitCode=0
Oct 30 00:40:19.588112 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588087    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerID="dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484" exitCode=0
Oct 30 00:40:19.588387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588116    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerDied","Data":"905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f"}
Oct 30 00:40:19.588387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588169    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerDied","Data":"94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd"}
Oct 30 00:40:19.588387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588196    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerDied","Data":"e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6"}
Oct 30 00:40:19.588387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588220    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerDied","Data":"f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801"}
Oct 30 00:40:19.588387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:19.588266    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerDied","Data":"dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484"}
Oct 30 00:40:20.543305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.543281    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:20.593847 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.593817    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerID="645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af" exitCode=0
Oct 30 00:40:20.593994 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.593860    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerDied","Data":"645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af"}
Oct 30 00:40:20.593994 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.593890    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:20.593994 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.593918    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6d29c6ab-8547-4cea-8828-24da7d652f22","Type":"ContainerDied","Data":"afecb9f745188d00f3d4a9d56c4ae04bac75cb914144de7a6e697c90b601ae3b"}
Oct 30 00:40:20.593994 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.593963    4576 scope.go:122] "RemoveContainer" containerID="905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f"
Oct 30 00:40:20.601420 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.601401    4576 scope.go:122] "RemoveContainer" containerID="94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd"
Oct 30 00:40:20.608216 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.608202    4576 scope.go:122] "RemoveContainer" containerID="645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af"
Oct 30 00:40:20.614689 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.614663    4576 scope.go:122] "RemoveContainer" containerID="e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6"
Oct 30 00:40:20.620940 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.620916    4576 scope.go:122] "RemoveContainer" containerID="f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801"
Oct 30 00:40:20.627117 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.627101    4576 scope.go:122] "RemoveContainer" containerID="dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484"
Oct 30 00:40:20.633932 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.633910    4576 scope.go:122] "RemoveContainer" containerID="56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9"
Oct 30 00:40:20.644647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.644622    4576 scope.go:122] "RemoveContainer" containerID="905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f"
Oct 30 00:40:20.645334 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:20.645304    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f\": container with ID starting with 905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f not found: ID does not exist" containerID="905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f"
Oct 30 00:40:20.645447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.645341    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f"} err="failed to get container status \"905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f\": rpc error: code = NotFound desc = could not find container \"905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f\": container with ID starting with 905b99a9f9cf4ce05f9f289e271f6990533e712a6ca3084f62cba04f6303700f not found: ID does not exist"
Oct 30 00:40:20.645447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.645384    4576 scope.go:122] "RemoveContainer" containerID="94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd"
Oct 30 00:40:20.645752 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:20.645728    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd\": container with ID starting with 94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd not found: ID does not exist" containerID="94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd"
Oct 30 00:40:20.645816 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.645760    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd"} err="failed to get container status \"94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd\": rpc error: code = NotFound desc = could not find container \"94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd\": container with ID starting with 94efdb54b015d217ee3e16886e85e66e10c098a9690ab0650452802505dfaccd not found: ID does not exist"
Oct 30 00:40:20.645816 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.645783    4576 scope.go:122] "RemoveContainer" containerID="645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af"
Oct 30 00:40:20.646105 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:20.646078    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af\": container with ID starting with 645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af not found: ID does not exist" containerID="645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af"
Oct 30 00:40:20.646181 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.646112    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af"} err="failed to get container status \"645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af\": rpc error: code = NotFound desc = could not find container \"645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af\": container with ID starting with 645e5aee07c66c759b95f817ecc86ae37e152a3574a7ecda137bbea9202455af not found: ID does not exist"
Oct 30 00:40:20.646181 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.646127    4576 scope.go:122] "RemoveContainer" containerID="e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6"
Oct 30 00:40:20.646501 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:20.646474    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6\": container with ID starting with e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6 not found: ID does not exist" containerID="e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6"
Oct 30 00:40:20.646618 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.646506    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6"} err="failed to get container status \"e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6\": rpc error: code = NotFound desc = could not find container \"e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6\": container with ID starting with e758b8d9d420e13383a581435b5455e24f084b9aafdd7908cb13b16ea34c23c6 not found: ID does not exist"
Oct 30 00:40:20.646618 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.646525    4576 scope.go:122] "RemoveContainer" containerID="f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801"
Oct 30 00:40:20.646867 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:20.646847    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801\": container with ID starting with f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801 not found: ID does not exist" containerID="f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801"
Oct 30 00:40:20.646956 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.646873    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801"} err="failed to get container status \"f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801\": rpc error: code = NotFound desc = could not find container \"f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801\": container with ID starting with f4eeacb698368cfbe73c5dc29c7be96dab242d59ac4949cb8d1b0bf9b9c5a801 not found: ID does not exist"
Oct 30 00:40:20.646956 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.646891    4576 scope.go:122] "RemoveContainer" containerID="dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484"
Oct 30 00:40:20.647176 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:20.647157    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484\": container with ID starting with dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484 not found: ID does not exist" containerID="dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484"
Oct 30 00:40:20.647280 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.647182    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484"} err="failed to get container status \"dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484\": rpc error: code = NotFound desc = could not find container \"dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484\": container with ID starting with dd27bb147aa25fcf0b118ce830b52f74a3cb2ab751ce039170d192f4f9a5a484 not found: ID does not exist"
Oct 30 00:40:20.647280 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.647202    4576 scope.go:122] "RemoveContainer" containerID="56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9"
Oct 30 00:40:20.647647 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:20.647624    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9\": container with ID starting with 56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9 not found: ID does not exist" containerID="56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9"
Oct 30 00:40:20.647747 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.647656    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9"} err="failed to get container status \"56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9\": rpc error: code = NotFound desc = could not find container \"56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9\": container with ID starting with 56672e3a88023caa6456bf636201bc0eda92b65b334e12f99350de811911caf9 not found: ID does not exist"
Oct 30 00:40:20.717216 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717185    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-metrics-client-certs\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717223    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-tls-assets\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-tls-assets\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717279    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-0\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717299    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-metrics-client-ca\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717318    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717339    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-1\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717358    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-kubelet-serving-ca-bundle\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717381    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-serving-certs-ca-bundle\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717409    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-2\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717448    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-tls\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717485    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-kube-api-access-t2xzh\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-kube-api-access-t2xzh\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717529    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-kube-rbac-proxy\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717575    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-trusted-ca-bundle\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717607    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717634    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-thanos-prometheus-http-client-file\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717656    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-config-out\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717682    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-db\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717715    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-web-config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-web-config\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717740    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-grpc-tls\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.717838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717776    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-config\") pod \"6d29c6ab-8547-4cea-8828-24da7d652f22\" (UID: \"6d29c6ab-8547-4cea-8828-24da7d652f22\") "
Oct 30 00:40:20.718624 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717837    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-metrics-client-ca" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:20.718624 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.717886    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-2" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:20.718624 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.718022    4576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-metrics-client-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.718624 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.718036    4576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.719037 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.719007    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-0" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:20.720137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.719116    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-1" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:20.720137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.719194    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-kubelet-serving-ca-bundle" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:20.720137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.719471    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-serving-certs-ca-bundle" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:20.720137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.719856    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-trusted-ca-bundle" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:40:20.720137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.720125    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-kube-rbac-proxy-web" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.720614 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.720169    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-metrics-client-certs" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.720614 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.720521    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-db" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:40:20.721713 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.721690    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-kube-rbac-proxy" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.722408 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.722384    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-tls-assets" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:20.722516 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.722454    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-config" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.723367 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.723343    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-thanos-prometheus-http-client-file" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.723672 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.723638    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-tls" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.723751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.723713    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-thanos-sidecar-tls" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.723751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.723736    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-grpc-tls" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.723917 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.723901    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-kube-api-access-t2xzh" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "kube-api-access-t2xzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:20.724191 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.724162    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-config-out" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:40:20.733201 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.733165    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-web-config" pod "6d29c6ab-8547-4cea-8828-24da7d652f22" (UID: "6d29c6ab-8547-4cea-8828-24da7d652f22"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:20.819385 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819352    4576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819385 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819379    4576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819385 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819393    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2xzh\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-kube-api-access-t2xzh\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819404    4576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-kube-rbac-proxy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819415    4576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819428    4576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819440    4576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819452    4576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-config-out\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819463    4576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-db\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819501    4576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-web-config\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819514    4576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-grpc-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819524    4576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-config\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819535    4576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-metrics-client-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819546    4576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6d29c6ab-8547-4cea-8828-24da7d652f22-tls-assets\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819557    4576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819569    4576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d29c6ab-8547-4cea-8828-24da7d652f22-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819583    4576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.819653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.819598    4576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d29c6ab-8547-4cea-8828-24da7d652f22-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:40:20.947293 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.947260    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:40:20.961401 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:20.961372    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:40:21.005998 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.005970    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:40:21.006255 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006226    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="init-config-reloader"
Oct 30 00:40:21.006306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006258    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="init-config-reloader"
Oct 30 00:40:21.006306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006267    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy-thanos"
Oct 30 00:40:21.006306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006272    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy-thanos"
Oct 30 00:40:21.006306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006281    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="thanos-sidecar"
Oct 30 00:40:21.006306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006286    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="thanos-sidecar"
Oct 30 00:40:21.006306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006294    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy"
Oct 30 00:40:21.006306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006298    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy"
Oct 30 00:40:21.006306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006307    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="prometheus"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006312    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="prometheus"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006318    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="config-reloader"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006322    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="config-reloader"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006328    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy-web"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006332    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy-web"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006339    4576 cpu_manager.go:414] "containerMap: removing container" podUID="97b48a54-79e8-44b4-8d9e-5d1046639884" containerName="insights-gathering"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006343    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b48a54-79e8-44b4-8d9e-5d1046639884" containerName="insights-gathering"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006386    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="prometheus"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006392    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="config-reloader"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006397    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy-web"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006402    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b48a54-79e8-44b4-8d9e-5d1046639884" containerName="insights-gathering"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006409    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy-thanos"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006414    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="thanos-sidecar"
Oct 30 00:40:21.006519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.006419    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" containerName="kube-rbac-proxy"
Oct 30 00:40:21.009854 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.009839    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.029327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.029290    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:40:21.029620 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.029602    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:40:21.029708 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.029608    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:40:21.029708 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.029623    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:40:21.029708 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.029699    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:40:21.029860 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.029849    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9qhgeickgd32l\""
Oct 30 00:40:21.030056 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.030044    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:40:21.030288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.030275    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:40:21.030479 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.030465    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wftph\""
Oct 30 00:40:21.030528 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.030516    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:40:21.030690 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.030676    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:40:21.032875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.032864    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:40:21.033019 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.033008    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:40:21.033189 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.033177    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 00:40:21.033265 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.033223    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:40:21.048202 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.048174    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:40:21.048469 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.048451    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:40:21.049089 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.049067    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:40:21.122362 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122329    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122362 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122361    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-config-out\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122609 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122378    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122609 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122392    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122609 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122408    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122609 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122563    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767lv\" (UniqueName: \"kubernetes.io/projected/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-kube-api-access-767lv\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122609 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122605    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122633    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122688    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-web-config\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122731    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122747    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122768    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122957 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122803    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122957 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122858    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122957 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122877    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122957 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122891    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122957 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122907    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.122957 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122925    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.123223 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.122973    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.123223 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.123057    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-config\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224055 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224028    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-config\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224059    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224078    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-config-out\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224093    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224113    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224138    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224169    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767lv\" (UniqueName: \"kubernetes.io/projected/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-kube-api-access-767lv\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224196    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224219    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224271    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-web-config\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224301    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224322    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224344    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224369    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224423    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224455    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224481    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224507    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224533    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.224573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224562    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.225136 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.224915    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.225645 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.225579    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.226751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.225726    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.226751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.226204    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.226751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.226412    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.228170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.227193    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-config-out\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.228170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.227642    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.228170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.227871    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.228170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.227894    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.228170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.227958    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.228170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.228028    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.228504 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.228423    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-config\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.228987 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.228966    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.229114 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.229090    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.229226 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.229209    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.229393 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.229371    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.230633 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.230613    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.230738 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.230722    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-web-config\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.230771 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.230755    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.246896 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.246877    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-767lv\" (UniqueName: \"kubernetes.io/projected/80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0-kube-api-access-767lv\") pod \"prometheus-k8s-0\" (UID: \"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.327247 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.327210    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:21.657380 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:21.657312    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:40:21.660326 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:40:21.660287    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fb8b2a_c3f2_4c42_bcfb_4a0d3f07fab0.slice/crio-f664524497165abd13b66baac7fe643678d074928027fa297a6099053e099e79 WatchSource:0}: Error finding container f664524497165abd13b66baac7fe643678d074928027fa297a6099053e099e79: Status 404 returned error can't find the container with id f664524497165abd13b66baac7fe643678d074928027fa297a6099053e099e79
Oct 30 00:40:22.104750 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:22.104715    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d29c6ab-8547-4cea-8828-24da7d652f22" path="/var/lib/kubelet/pods/6d29c6ab-8547-4cea-8828-24da7d652f22/volumes"
Oct 30 00:40:22.602168 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:22.602133    4576 generic.go:356] "Generic (PLEG): container finished" podID="80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0" containerID="42e10fac8ea35a169c07dfbbb9d95d10e1a6f9df7864810e8418d6289e5d2a59" exitCode=0
Oct 30 00:40:22.602340 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:22.602181    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0","Type":"ContainerDied","Data":"42e10fac8ea35a169c07dfbbb9d95d10e1a6f9df7864810e8418d6289e5d2a59"}
Oct 30 00:40:22.602340 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:22.602246    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0","Type":"ContainerStarted","Data":"f664524497165abd13b66baac7fe643678d074928027fa297a6099053e099e79"}
Oct 30 00:40:23.608947 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.608912    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0","Type":"ContainerStarted","Data":"f1f367afdbc6b6bc5be4144aae43c80ceaf7fcea464e1372e7303a035f7645f3"}
Oct 30 00:40:23.609354 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.608998    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0","Type":"ContainerStarted","Data":"0b0abbfcbb736316ab2d7c2f4a59753abc1640514fed57ea7c811b766d6205f7"}
Oct 30 00:40:23.609354 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.609031    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0","Type":"ContainerStarted","Data":"46404859b5eb20c1778171a34d3013c7d5cd1e8b293a46c8eb3cffb1efb36aa5"}
Oct 30 00:40:23.609354 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.609057    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0","Type":"ContainerStarted","Data":"81f151c50e69d4b5c3d3e19feeb18eb276322bbaefc1cef9655005280f0e0afe"}
Oct 30 00:40:23.609354 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.609078    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0","Type":"ContainerStarted","Data":"2bbadec5c007e8672b5b8d5c532d583b4d10a5cce146a99972c197dab6150c11"}
Oct 30 00:40:23.609354 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.609102    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0","Type":"ContainerStarted","Data":"f5e2f495198cdf69c109bcd2372d2a2ab95f13f61f19129a8a711b14cf73348e"}
Oct 30 00:40:23.616843 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.616813    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:23.700566 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.700518    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.684998237 podStartE2EDuration="3.700505188s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.506951ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:23.686393117 +0000 UTC m=+103.757716955" watchObservedRunningTime="2026-10-30 00:40:23.700505188 +0000 UTC m=+103.771829018"
Oct 30 00:40:23.759640 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:23.759609    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:25.736402 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:25.736358    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:26.327355 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:26.327317    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:40:30.065759 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:30.065719    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:30.259847 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:30.259803    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:30.425497 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:30.425420    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:34.696345 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:34.696303    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:34.817647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:34.817607    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:38.753849 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:40:38.753806    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:40:48.379855 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:40:48.379815    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:41:18.381920 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:41:18.381836    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:41:21.327979 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:41:21.327948    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:41:21.343944 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:41:21.343922    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:41:21.827020 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:41:21.826987    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:41:25.938831 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:41:25.938796    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:41:34.423339 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:41:34.423301    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:41:48.378956 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:41:48.378917    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:42:11.390103 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:11.390056    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:18.380389 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:42:18.380350    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:42:25.936632 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:25.936543    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:27.462555 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:27.462494    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:27.584447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:27.584410    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:37.967103 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:37.967052    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:38.096967 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:38.096933    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:43.967403 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:43.967364    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:44.080677 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:44.080643    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:45.740878 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:45.740842    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:45.850999 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:45.850965    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:46.917569 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:46.917529    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:47.036874 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:47.036842    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:47.629433 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:47.629398    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:47.749518 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:47.749480    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:42:48.380403 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:42:48.380366    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:42:59.989252 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:42:59.989191    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:01.347253 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:01.347199    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:03.200544 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:03.200514    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:04.389063 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:04.389026    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:06.358942 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:06.358910    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:06.516464 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:06.516431    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:06.685808 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:06.685722    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:08.181273 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:08.181220    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:08.743574 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:08.743540    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:10.192085 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:10.192050    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:10.301899 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:10.301856    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:17.587912 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:17.587869    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:17.808603 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:17.808558    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:17.920087 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:17.920004    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:18.379072 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:43:18.379039    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:43:21.690363 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:21.690313    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:21.816088 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:21.816057    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:24.186125 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:24.186096    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:24.316774 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:24.316740    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:26.046933 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:26.046898    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:26.154503 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:26.154455    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:36.601643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:36.601609    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:36.801455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:36.801412    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:36.932711 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:36.932616    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:37.133195 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.133151    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:37.291573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.291543    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"]
Oct 30 00:43:37.294708 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.294688    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.334312 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.334283    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"]
Oct 30 00:43:37.395357 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.395323    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 00:43:37.395357 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.395331    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 00:43:37.395976 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.395958    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 00:43:37.396042 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.396030    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f4n97\""
Oct 30 00:43:37.396146 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.396130    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 00:43:37.396178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.396160    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 00:43:37.409689 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.409658    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:37.467580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.467555    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.467733 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.467586    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-metrics-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.467733 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.467606    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.467733 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.467685    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e93e8d8-9c70-446a-ad48-ead2f959237a-tmp\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.467860 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.467728    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gnq\" (UniqueName: \"kubernetes.io/projected/2e93e8d8-9c70-446a-ad48-ead2f959237a-kube-api-access-84gnq\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.467860 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.467797    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e93e8d8-9c70-446a-ad48-ead2f959237a-manifests\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.551979 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.551890    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:37.568394 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.568372    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e93e8d8-9c70-446a-ad48-ead2f959237a-manifests\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.568495 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.568417    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.568495 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.568438    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-metrics-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.568495 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.568455    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.568495 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.568480    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e93e8d8-9c70-446a-ad48-ead2f959237a-tmp\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.568686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.568496    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gnq\" (UniqueName: \"kubernetes.io/projected/2e93e8d8-9c70-446a-ad48-ead2f959237a-kube-api-access-84gnq\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.568875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.568851    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e93e8d8-9c70-446a-ad48-ead2f959237a-manifests\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.571071 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.571043    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e93e8d8-9c70-446a-ad48-ead2f959237a-tmp\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.571184 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.571121    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.571184 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.571159    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-metrics-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.571290 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.571247    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e93e8d8-9c70-446a-ad48-ead2f959237a-webhook-cert\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.591625 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.590325    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gnq\" (UniqueName: \"kubernetes.io/projected/2e93e8d8-9c70-446a-ad48-ead2f959237a-kube-api-access-84gnq\") pod \"opendatahub-operator-controller-manager-8cc7fc69d-7kw5f\" (UID: \"2e93e8d8-9c70-446a-ad48-ead2f959237a\") " pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.608981 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.608955    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:37.666110 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.666076    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:37.960063 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:37.959892    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"]
Oct 30 00:43:37.962902 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:43:37.962874    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e93e8d8_9c70_446a_ad48_ead2f959237a.slice/crio-d521d4ae32ff4ccb361ff5dca8704e275ab3c1f384a34f4250662d79da3f3a38 WatchSource:0}: Error finding container d521d4ae32ff4ccb361ff5dca8704e275ab3c1f384a34f4250662d79da3f3a38: Status 404 returned error can't find the container with id d521d4ae32ff4ccb361ff5dca8704e275ab3c1f384a34f4250662d79da3f3a38
Oct 30 00:43:38.246991 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:38.246914    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f" event={"ID":"2e93e8d8-9c70-446a-ad48-ead2f959237a","Type":"ContainerStarted","Data":"d521d4ae32ff4ccb361ff5dca8704e275ab3c1f384a34f4250662d79da3f3a38"}
Oct 30 00:43:38.691797 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:38.691756    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:41.259876 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:41.259842    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f" event={"ID":"2e93e8d8-9c70-446a-ad48-ead2f959237a","Type":"ContainerStarted","Data":"d30a8a590857a4433827ae32921b31fa41b2e4e94843feb11a648ecb79df74be"}
Oct 30 00:43:42.264677 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:42.264641    4576 generic.go:356] "Generic (PLEG): container finished" podID="2e93e8d8-9c70-446a-ad48-ead2f959237a" containerID="d30a8a590857a4433827ae32921b31fa41b2e4e94843feb11a648ecb79df74be" exitCode=0
Oct 30 00:43:42.265165 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:42.264704    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f" event={"ID":"2e93e8d8-9c70-446a-ad48-ead2f959237a","Type":"ContainerDied","Data":"d30a8a590857a4433827ae32921b31fa41b2e4e94843feb11a648ecb79df74be"}
Oct 30 00:43:42.265400 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:42.265364    4576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:43:42.557183 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:42.557102    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:42.676283 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:42.676253    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:43.270386 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:43.270354    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f" event={"ID":"2e93e8d8-9c70-446a-ad48-ead2f959237a","Type":"ContainerStarted","Data":"f998d0f35c9d42fa3a041a1c5e368f4a73a74d5db3ce5333fb3e63f3a4c24a45"}
Oct 30 00:43:43.270774 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:43.270505    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:43.353014 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:43.352966    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f" podStartSLOduration=2.7579195199999997 podStartE2EDuration="6.352952477s" totalImagesPullingTime="3.103528538s" totalInitContainerRuntime="491.504419ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:43.338616321 +0000 UTC m=+303.409940155" watchObservedRunningTime="2026-10-30 00:43:43.352952477 +0000 UTC m=+303.424276311"
Oct 30 00:43:48.380037 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:43:48.379995    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:43:54.277439 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:54.277343    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8cc7fc69d-7kw5f"
Oct 30 00:43:56.658258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:56.658200    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:56.797640 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:56.797603    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:57.108189 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:57.108141    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:57.305178 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:57.305143    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:58.367255 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:58.367200    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:58.515677 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:58.515632    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:59.171419 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.171390    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"]
Oct 30 00:43:59.176130 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.176110    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.203995 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.203971    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"]
Oct 30 00:43:59.216498 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.216478    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 00:43:59.216623 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.216504    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:43:59.216623 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.216518    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 00:43:59.216704 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.216616    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 00:43:59.216857 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.216846    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 00:43:59.217141 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.217126    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bdjc8\""
Oct 30 00:43:59.232568 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.232541    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-manager-config\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.232658 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.232577    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-cert\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.232658 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.232606    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-metrics-cert\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.232777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.232759    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65dp\" (UniqueName: \"kubernetes.io/projected/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-kube-api-access-s65dp\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.334024 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.333990    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65dp\" (UniqueName: \"kubernetes.io/projected/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-kube-api-access-s65dp\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.334320 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.334042    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-manager-config\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.334320 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.334081    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-cert\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.334320 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.334122    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-metrics-cert\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.334771 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.334753    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-manager-config\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.336679 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.336659    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-metrics-cert\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.336776 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.336758    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-cert\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.360331 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.360294    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65dp\" (UniqueName: \"kubernetes.io/projected/4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682-kube-api-access-s65dp\") pod \"lws-controller-manager-7459cbd475-shmmv\" (UID: \"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682\") " pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.486031 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.485992    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:43:59.614709 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.614674    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:43:59.780364 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.780293    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"]
Oct 30 00:43:59.783592 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:43:59.783569    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7cd1d3_d468_4ac1_918d_ec1b2d3e1682.slice/crio-e40a8421b475ab3408a447707bdc64c377e71073e366cb093c602d788f299303 WatchSource:0}: Error finding container e40a8421b475ab3408a447707bdc64c377e71073e366cb093c602d788f299303: Status 404 returned error can't find the container with id e40a8421b475ab3408a447707bdc64c377e71073e366cb093c602d788f299303
Oct 30 00:43:59.853215 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:43:59.853183    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:00.088362 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.088295    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-6d7mf"]
Oct 30 00:44:00.092477 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.092462    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:00.108776 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.108749    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 00:44:00.108927 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.108840    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-x4tgq\""
Oct 30 00:44:00.123862 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.123839    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-6d7mf"]
Oct 30 00:44:00.140513 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.140480    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59px\" (UniqueName: \"kubernetes.io/projected/d049c7b9-2cae-42e9-b404-fb375a91a2cf-kube-api-access-n59px\") pod \"odh-observability-584f99c788-6d7mf\" (UID: \"d049c7b9-2cae-42e9-b404-fb375a91a2cf\") " pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:00.140650 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.140528    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/d049c7b9-2cae-42e9-b404-fb375a91a2cf-webhook-certs\") pod \"odh-observability-584f99c788-6d7mf\" (UID: \"d049c7b9-2cae-42e9-b404-fb375a91a2cf\") " pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:00.219571 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.219530    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:00.241686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.241657    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59px\" (UniqueName: \"kubernetes.io/projected/d049c7b9-2cae-42e9-b404-fb375a91a2cf-kube-api-access-n59px\") pod \"odh-observability-584f99c788-6d7mf\" (UID: \"d049c7b9-2cae-42e9-b404-fb375a91a2cf\") " pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:00.241827 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.241709    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/d049c7b9-2cae-42e9-b404-fb375a91a2cf-webhook-certs\") pod \"odh-observability-584f99c788-6d7mf\" (UID: \"d049c7b9-2cae-42e9-b404-fb375a91a2cf\") " pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:00.241827 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:44:00.241801    4576 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 00:44:00.241918 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:44:00.241851    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d049c7b9-2cae-42e9-b404-fb375a91a2cf-webhook-certs podName:d049c7b9-2cae-42e9-b404-fb375a91a2cf nodeName:}" failed. No retries permitted until 2026-10-30 00:44:00.741836366 +0000 UTC m=+320.813160179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/d049c7b9-2cae-42e9-b404-fb375a91a2cf-webhook-certs") pod "odh-observability-584f99c788-6d7mf" (UID: "d049c7b9-2cae-42e9-b404-fb375a91a2cf") : secret "odh-observability-webhook-cert" not found
Oct 30 00:44:00.263907 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.263876    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59px\" (UniqueName: \"kubernetes.io/projected/d049c7b9-2cae-42e9-b404-fb375a91a2cf-kube-api-access-n59px\") pod \"odh-observability-584f99c788-6d7mf\" (UID: \"d049c7b9-2cae-42e9-b404-fb375a91a2cf\") " pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:00.333131 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.333099    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv" event={"ID":"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682","Type":"ContainerStarted","Data":"e40a8421b475ab3408a447707bdc64c377e71073e366cb093c602d788f299303"}
Oct 30 00:44:00.533161 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.533128    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:00.746202 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.746169    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/d049c7b9-2cae-42e9-b404-fb375a91a2cf-webhook-certs\") pod \"odh-observability-584f99c788-6d7mf\" (UID: \"d049c7b9-2cae-42e9-b404-fb375a91a2cf\") " pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:00.748853 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:00.748828    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/d049c7b9-2cae-42e9-b404-fb375a91a2cf-webhook-certs\") pod \"odh-observability-584f99c788-6d7mf\" (UID: \"d049c7b9-2cae-42e9-b404-fb375a91a2cf\") " pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:01.004968 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:01.004923    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:01.311908 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:01.311833    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-6d7mf"]
Oct 30 00:44:01.316016 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:44:01.315987    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd049c7b9_2cae_42e9_b404_fb375a91a2cf.slice/crio-eb7060a1cc342a6c4c690919d90daadc481ac6181cf512eab1657dd94e381875 WatchSource:0}: Error finding container eb7060a1cc342a6c4c690919d90daadc481ac6181cf512eab1657dd94e381875: Status 404 returned error can't find the container with id eb7060a1cc342a6c4c690919d90daadc481ac6181cf512eab1657dd94e381875
Oct 30 00:44:01.338390 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:01.338355    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-6d7mf" event={"ID":"d049c7b9-2cae-42e9-b404-fb375a91a2cf","Type":"ContainerStarted","Data":"eb7060a1cc342a6c4c690919d90daadc481ac6181cf512eab1657dd94e381875"}
Oct 30 00:44:03.354361 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:03.354321    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv" event={"ID":"4d7cd1d3-d468-4ac1-918d-ec1b2d3e1682","Type":"ContainerStarted","Data":"bf8fd2892dce18c931dc9cea8071b009ad2a878b22c56770b31bfffdf634e763"}
Oct 30 00:44:03.354780 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:03.354493    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:44:03.397155 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:03.397090    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv" podStartSLOduration=1.535476557 podStartE2EDuration="4.397072103s" totalImagesPullingTime="2.861595546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:03.396111529 +0000 UTC m=+323.467435396" watchObservedRunningTime="2026-10-30 00:44:03.397072103 +0000 UTC m=+323.468395940"
Oct 30 00:44:04.358914 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:04.358874    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-6d7mf" event={"ID":"d049c7b9-2cae-42e9-b404-fb375a91a2cf","Type":"ContainerStarted","Data":"638f1bb83c7c14d6d0afa7bf40f5e5ea6378c12ab21573c097b64338ca564ea8"}
Oct 30 00:44:04.359455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:04.359046    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:11.641130 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.641092    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:11.692171 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.692102    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-6d7mf" podStartSLOduration=9.385127066999999 podStartE2EDuration="11.692083746s" totalImagesPullingTime="2.306956679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:04.414258473 +0000 UTC m=+324.485582309" watchObservedRunningTime="2026-10-30 00:44:11.692083746 +0000 UTC m=+331.763407582"
Oct 30 00:44:11.693093 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.693077    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"]
Oct 30 00:44:11.696500 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.696486    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.714142 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.714123    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-228fl\""
Oct 30 00:44:11.728163 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.728125    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"]
Oct 30 00:44:11.733569 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.733550    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fadd188-7027-434e-9d8f-f9d63a62ef6b-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.733662 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.733582    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fadd188-7027-434e-9d8f-f9d63a62ef6b-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.733721 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.733707    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pptc\" (UniqueName: \"kubernetes.io/projected/8fadd188-7027-434e-9d8f-f9d63a62ef6b-kube-api-access-6pptc\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.733752 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.733737    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8fadd188-7027-434e-9d8f-f9d63a62ef6b-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.756280 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.756259    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 00:44:11.834297 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.834258    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pptc\" (UniqueName: \"kubernetes.io/projected/8fadd188-7027-434e-9d8f-f9d63a62ef6b-kube-api-access-6pptc\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.834459 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.834308    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8fadd188-7027-434e-9d8f-f9d63a62ef6b-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.834459 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.834345    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fadd188-7027-434e-9d8f-f9d63a62ef6b-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.834459 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.834363    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fadd188-7027-434e-9d8f-f9d63a62ef6b-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.834695 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.834675    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8fadd188-7027-434e-9d8f-f9d63a62ef6b-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.834765 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.834702    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fadd188-7027-434e-9d8f-f9d63a62ef6b-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.835207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.835183    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8fadd188-7027-434e-9d8f-f9d63a62ef6b-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:11.853225 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.853200    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:11.856659 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:11.856638    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pptc\" (UniqueName: \"kubernetes.io/projected/8fadd188-7027-434e-9d8f-f9d63a62ef6b-kube-api-access-6pptc\") pod \"kserve-module-controller-manager-5cfb77bf8c-gkxfc\" (UID: \"8fadd188-7027-434e-9d8f-f9d63a62ef6b\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:12.008911 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:12.008876    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:12.314825 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:12.314577    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"]
Oct 30 00:44:12.317221 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:44:12.317195    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fadd188_7027_434e_9d8f_f9d63a62ef6b.slice/crio-1db8d54039256c907d6f413472e8041679a3efaa0b927b92cb444b71c4bbb156 WatchSource:0}: Error finding container 1db8d54039256c907d6f413472e8041679a3efaa0b927b92cb444b71c4bbb156: Status 404 returned error can't find the container with id 1db8d54039256c907d6f413472e8041679a3efaa0b927b92cb444b71c4bbb156
Oct 30 00:44:12.387240 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:12.387208    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc" event={"ID":"8fadd188-7027-434e-9d8f-f9d63a62ef6b","Type":"ContainerStarted","Data":"1db8d54039256c907d6f413472e8041679a3efaa0b927b92cb444b71c4bbb156"}
Oct 30 00:44:14.361863 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:14.361834    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7459cbd475-shmmv"
Oct 30 00:44:15.364778 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:15.364750    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-6d7mf"
Oct 30 00:44:15.400221 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:15.400185    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc" event={"ID":"8fadd188-7027-434e-9d8f-f9d63a62ef6b","Type":"ContainerStarted","Data":"2597005f138aa17269d9e5e291118bcb2c36bb48f1ea4ecb708848821c258235"}
Oct 30 00:44:15.400389 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:15.400312    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:15.449206 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:15.449148    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc" podStartSLOduration=2.208518618 podStartE2EDuration="4.44912744s" totalImagesPullingTime="2.240608822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:15.448188257 +0000 UTC m=+335.519512092" watchObservedRunningTime="2026-10-30 00:44:15.44912744 +0000 UTC m=+335.520451275"
Oct 30 00:44:16.330682 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:16.330645    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:16.436904 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:16.436868    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:16.981538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:16.981503    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:17.107331 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:17.107293    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:17.637338 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:17.637291    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:17.875614 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:17.875576    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:18.011859 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:18.011823    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:18.379553 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:44:18.379467    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:44:26.156046 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:26.156014    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:36.467821 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:36.467792    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-gkxfc"
Oct 30 00:44:48.379954 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:44:48.379912    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:44:58.858042 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:58.858000    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:59.014772 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.014720    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:59.130739 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.130659    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hp8xp"]
Oct 30 00:44:59.137361 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.137334    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:44:59.153074 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.153051    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hp8xp"]
Oct 30 00:44:59.169562 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.169540    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hbbbj\""
Oct 30 00:44:59.169562 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.169540    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 00:44:59.170509 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.170492    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"]
Oct 30 00:44:59.173655 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.173635    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:44:59.174444 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.174425    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:59.192709 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.192683    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"]
Oct 30 00:44:59.222445 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.222418    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x78xr"]
Oct 30 00:44:59.226910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.226888    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:44:59.250752 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.250729    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x78xr"]
Oct 30 00:44:59.259195 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.259169    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 00:44:59.259393 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.259374    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-l29rt\""
Oct 30 00:44:59.259476 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.259417    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vvnw7\""
Oct 30 00:44:59.259476 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.259426    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 00:44:59.261468 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.261447    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zkgm7"]
Oct 30 00:44:59.266321 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.266303    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.282643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.282621    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zkgm7"]
Oct 30 00:44:59.305356 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.305311    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:59.327994 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.327972    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1592b36f-bac4-4be4-ab7b-bbbe440f51da-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xz2hh\" (UID: \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:44:59.328104 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.328002    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4kl\" (UniqueName: \"kubernetes.io/projected/61283cde-12bc-48f5-8bda-faa1da468ca7-kube-api-access-kq4kl\") pod \"kserve-controller-manager-776cc66d47-hp8xp\" (UID: \"61283cde-12bc-48f5-8bda-faa1da468ca7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:44:59.328104 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.328028    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c70240-814b-4be3-9c3e-3bc79a889b38-tls-certs\") pod \"model-serving-api-75bd89fdfc-x78xr\" (UID: \"97c70240-814b-4be3-9c3e-3bc79a889b38\") " pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:44:59.328104 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.328084    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61283cde-12bc-48f5-8bda-faa1da468ca7-cert\") pod \"kserve-controller-manager-776cc66d47-hp8xp\" (UID: \"61283cde-12bc-48f5-8bda-faa1da468ca7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:44:59.328204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.328153    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvr9\" (UniqueName: \"kubernetes.io/projected/97c70240-814b-4be3-9c3e-3bc79a889b38-kube-api-access-4tvr9\") pod \"model-serving-api-75bd89fdfc-x78xr\" (UID: \"97c70240-814b-4be3-9c3e-3bc79a889b38\") " pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:44:59.328204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.328176    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfj76\" (UniqueName: \"kubernetes.io/projected/1592b36f-bac4-4be4-ab7b-bbbe440f51da-kube-api-access-bfj76\") pod \"llmisvc-controller-manager-869d5dcc5d-xz2hh\" (UID: \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:44:59.360214 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.360192    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-88tq8\""
Oct 30 00:44:59.360214 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.360210    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 00:44:59.360422 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.360193    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 00:44:59.429053 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.428969    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22q2\" (UniqueName: \"kubernetes.io/projected/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-kube-api-access-g22q2\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.429253 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.429091    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvr9\" (UniqueName: \"kubernetes.io/projected/97c70240-814b-4be3-9c3e-3bc79a889b38-kube-api-access-4tvr9\") pod \"model-serving-api-75bd89fdfc-x78xr\" (UID: \"97c70240-814b-4be3-9c3e-3bc79a889b38\") " pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:44:59.429253 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.429140    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-cert\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.429253 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.429169    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfj76\" (UniqueName: \"kubernetes.io/projected/1592b36f-bac4-4be4-ab7b-bbbe440f51da-kube-api-access-bfj76\") pod \"llmisvc-controller-manager-869d5dcc5d-xz2hh\" (UID: \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:44:59.429437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.429226    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1592b36f-bac4-4be4-ab7b-bbbe440f51da-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xz2hh\" (UID: \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:44:59.429437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.429282    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4kl\" (UniqueName: \"kubernetes.io/projected/61283cde-12bc-48f5-8bda-faa1da468ca7-kube-api-access-kq4kl\") pod \"kserve-controller-manager-776cc66d47-hp8xp\" (UID: \"61283cde-12bc-48f5-8bda-faa1da468ca7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:44:59.429437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.429301    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.429437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.429342    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c70240-814b-4be3-9c3e-3bc79a889b38-tls-certs\") pod \"model-serving-api-75bd89fdfc-x78xr\" (UID: \"97c70240-814b-4be3-9c3e-3bc79a889b38\") " pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:44:59.429437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.429402    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61283cde-12bc-48f5-8bda-faa1da468ca7-cert\") pod \"kserve-controller-manager-776cc66d47-hp8xp\" (UID: \"61283cde-12bc-48f5-8bda-faa1da468ca7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:44:59.432418 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.432386    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1592b36f-bac4-4be4-ab7b-bbbe440f51da-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xz2hh\" (UID: \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:44:59.432663 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.432645    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61283cde-12bc-48f5-8bda-faa1da468ca7-cert\") pod \"kserve-controller-manager-776cc66d47-hp8xp\" (UID: \"61283cde-12bc-48f5-8bda-faa1da468ca7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:44:59.432800 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.432779    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c70240-814b-4be3-9c3e-3bc79a889b38-tls-certs\") pod \"model-serving-api-75bd89fdfc-x78xr\" (UID: \"97c70240-814b-4be3-9c3e-3bc79a889b38\") " pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:44:59.450125 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.450097    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4kl\" (UniqueName: \"kubernetes.io/projected/61283cde-12bc-48f5-8bda-faa1da468ca7-kube-api-access-kq4kl\") pod \"kserve-controller-manager-776cc66d47-hp8xp\" (UID: \"61283cde-12bc-48f5-8bda-faa1da468ca7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:44:59.450271 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.450159    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvr9\" (UniqueName: \"kubernetes.io/projected/97c70240-814b-4be3-9c3e-3bc79a889b38-kube-api-access-4tvr9\") pod \"model-serving-api-75bd89fdfc-x78xr\" (UID: \"97c70240-814b-4be3-9c3e-3bc79a889b38\") " pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:44:59.450370 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.450354    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfj76\" (UniqueName: \"kubernetes.io/projected/1592b36f-bac4-4be4-ab7b-bbbe440f51da-kube-api-access-bfj76\") pod \"llmisvc-controller-manager-869d5dcc5d-xz2hh\" (UID: \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:44:59.486303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.486273    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:44:59.529807 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.529774    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-cert\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.529960 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.529825    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.529960 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.529874    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22q2\" (UniqueName: \"kubernetes.io/projected/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-kube-api-access-g22q2\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.532296 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.532272    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-metrics-certs\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.532818 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.532798    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-cert\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.544645 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.544596    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:44:59.552933 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.552907    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22q2\" (UniqueName: \"kubernetes.io/projected/0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f-kube-api-access-g22q2\") pod \"odh-model-controller-f87fcf56c-zkgm7\" (UID: \"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f\") " pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.580348 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.580320    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:44:59.616003 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.615905    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:44:59.749300 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.749275    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:44:59.784604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.784581    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"]
Oct 30 00:44:59.787785 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:44:59.787752    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1592b36f_bac4_4be4_ab7b_bbbe440f51da.slice/crio-22e38e02d02c7f5def5f85aabf204babc31dee99e52f52c1982a1d6fa3c599f5 WatchSource:0}: Error finding container 22e38e02d02c7f5def5f85aabf204babc31dee99e52f52c1982a1d6fa3c599f5: Status 404 returned error can't find the container with id 22e38e02d02c7f5def5f85aabf204babc31dee99e52f52c1982a1d6fa3c599f5
Oct 30 00:44:59.843740 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.843697    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x78xr"]
Oct 30 00:44:59.845686 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:44:59.845659    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c70240_814b_4be3_9c3e_3bc79a889b38.slice/crio-c6d71b1e41e2d2a1357195339a7b8a751e2a2a8ed373fcd3a3e537ef073223c9 WatchSource:0}: Error finding container c6d71b1e41e2d2a1357195339a7b8a751e2a2a8ed373fcd3a3e537ef073223c9: Status 404 returned error can't find the container with id c6d71b1e41e2d2a1357195339a7b8a751e2a2a8ed373fcd3a3e537ef073223c9
Oct 30 00:44:59.884887 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:44:59.884860    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-zkgm7"]
Oct 30 00:44:59.887506 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:44:59.887482    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9f28eb_aba2_47ed_a91b_43b7cf2ec71f.slice/crio-9339ac1d81bfe280a39c45c55d10fc4041aa0efaad4045056aa364283aaa90f3 WatchSource:0}: Error finding container 9339ac1d81bfe280a39c45c55d10fc4041aa0efaad4045056aa364283aaa90f3: Status 404 returned error can't find the container with id 9339ac1d81bfe280a39c45c55d10fc4041aa0efaad4045056aa364283aaa90f3
Oct 30 00:45:00.046563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:00.046497    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hp8xp"]
Oct 30 00:45:00.049609 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:45:00.049574    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61283cde_12bc_48f5_8bda_faa1da468ca7.slice/crio-3cfedbe2d260de0c86062a747aa7ef463c5b280d6540d4096505c4eae56c843f WatchSource:0}: Error finding container 3cfedbe2d260de0c86062a747aa7ef463c5b280d6540d4096505c4eae56c843f: Status 404 returned error can't find the container with id 3cfedbe2d260de0c86062a747aa7ef463c5b280d6540d4096505c4eae56c843f
Oct 30 00:45:00.568742 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:00.568700    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7" event={"ID":"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f","Type":"ContainerStarted","Data":"9339ac1d81bfe280a39c45c55d10fc4041aa0efaad4045056aa364283aaa90f3"}
Oct 30 00:45:00.571137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:00.571099    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp" event={"ID":"61283cde-12bc-48f5-8bda-faa1da468ca7","Type":"ContainerStarted","Data":"3cfedbe2d260de0c86062a747aa7ef463c5b280d6540d4096505c4eae56c843f"}
Oct 30 00:45:00.574882 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:00.574853    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-x78xr" event={"ID":"97c70240-814b-4be3-9c3e-3bc79a889b38","Type":"ContainerStarted","Data":"c6d71b1e41e2d2a1357195339a7b8a751e2a2a8ed373fcd3a3e537ef073223c9"}
Oct 30 00:45:00.576529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:00.576505    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh" event={"ID":"1592b36f-bac4-4be4-ab7b-bbbe440f51da","Type":"ContainerStarted","Data":"22e38e02d02c7f5def5f85aabf204babc31dee99e52f52c1982a1d6fa3c599f5"}
Oct 30 00:45:01.426150 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:01.426081    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:45:01.553679 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:01.553625    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:45:01.680930 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:01.680778    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:45:01.800769 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:01.800737    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:45:02.588247 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:02.588207    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-x78xr" event={"ID":"97c70240-814b-4be3-9c3e-3bc79a889b38","Type":"ContainerStarted","Data":"282d3effd0138f6f06f4424ce2ae1e1bd08af912a3d12c1ee467091bd36b8321"}
Oct 30 00:45:02.588669 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:02.588377    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:45:02.631394 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:02.631331    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-x78xr" podStartSLOduration=0.989862556 podStartE2EDuration="3.631313836s" totalImagesPullingTime="2.64145128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:02.630513078 +0000 UTC m=+382.701836911" watchObservedRunningTime="2026-10-30 00:45:02.631313836 +0000 UTC m=+382.702637715"
Oct 30 00:45:03.598290 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.598254    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7" event={"ID":"0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f","Type":"ContainerStarted","Data":"89e1a7c5b03375853b9dd5d90df516e6e98254e8b87bc0121079157fcf470e17"}
Oct 30 00:45:03.598773 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.598354    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:45:03.599857 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.599832    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp" event={"ID":"61283cde-12bc-48f5-8bda-faa1da468ca7","Type":"ContainerStarted","Data":"768fd7e14f1be60ee201bc98eaafa8dbcfd0e24c5bf8d9c9c7f4d04ec445a69d"}
Oct 30 00:45:03.600070 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.599895    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:45:03.601493 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.601468    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh" event={"ID":"1592b36f-bac4-4be4-ab7b-bbbe440f51da","Type":"ContainerStarted","Data":"baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954"}
Oct 30 00:45:03.601625 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.601612    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:45:03.639606 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.639521    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7" podStartSLOduration=1.166218565 podStartE2EDuration="4.639510221s" totalImagesPullingTime="3.473291656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:03.639354052 +0000 UTC m=+383.710677886" watchObservedRunningTime="2026-10-30 00:45:03.639510221 +0000 UTC m=+383.710834054"
Oct 30 00:45:03.681017 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.680972    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp" podStartSLOduration=1.815002534 podStartE2EDuration="4.680959478s" totalImagesPullingTime="2.865956944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:03.680183139 +0000 UTC m=+383.751506974" watchObservedRunningTime="2026-10-30 00:45:03.680959478 +0000 UTC m=+383.752283312"
Oct 30 00:45:03.720834 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:03.720789    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh" podStartSLOduration=1.968568949 podStartE2EDuration="4.720776394s" totalImagesPullingTime="2.752207445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:03.720473264 +0000 UTC m=+383.791797093" watchObservedRunningTime="2026-10-30 00:45:03.720776394 +0000 UTC m=+383.792100228"
Oct 30 00:45:09.382621 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:09.382583    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:45:09.497877 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:09.497839    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:45:13.606717 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:13.606689    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-x78xr"
Oct 30 00:45:14.608522 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:14.608489    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-zkgm7"
Oct 30 00:45:18.380367 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:45:18.380333    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:45:26.262710 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:26.262632    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:45:34.608924 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:34.608892    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:45:34.611794 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:34.611774    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hp8xp"
Oct 30 00:45:48.380083 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:45:48.380039    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:45:50.030902 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:50.030862    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:45:50.148414 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:45:50.148378    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:02.621651 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:02.621608    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:03.420757 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:03.420719    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:04.207724 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:04.207686    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:08.124670 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:08.124626    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:09.631253 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:09.631198    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:10.036626 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:10.036591    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:10.299086 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:10.298993    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:11.375214 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:11.375182    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:11.516193 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:11.516160    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:11.996388 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:11.996346    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-jzffv"]
Oct 30 00:46:11.998829 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:11.998813    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv"
Oct 30 00:46:12.015120 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.015093    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 00:46:12.029160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.029124    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 00:46:12.029368 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.029298    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-crb9v\""
Oct 30 00:46:12.031204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.031183    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-jzffv"]
Oct 30 00:46:12.036577 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.036552    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrk7z\" (UniqueName: \"kubernetes.io/projected/c7b2a498-5dc3-4d75-9859-f064b91e522f-kube-api-access-lrk7z\") pod \"authorino-operator-68d6dd76dc-jzffv\" (UID: \"c7b2a498-5dc3-4d75-9859-f064b91e522f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv"
Oct 30 00:46:12.138022 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.137983    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrk7z\" (UniqueName: \"kubernetes.io/projected/c7b2a498-5dc3-4d75-9859-f064b91e522f-kube-api-access-lrk7z\") pod \"authorino-operator-68d6dd76dc-jzffv\" (UID: \"c7b2a498-5dc3-4d75-9859-f064b91e522f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv"
Oct 30 00:46:12.159057 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.159027    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrk7z\" (UniqueName: \"kubernetes.io/projected/c7b2a498-5dc3-4d75-9859-f064b91e522f-kube-api-access-lrk7z\") pod \"authorino-operator-68d6dd76dc-jzffv\" (UID: \"c7b2a498-5dc3-4d75-9859-f064b91e522f\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv"
Oct 30 00:46:12.310748 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.310662    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv"
Oct 30 00:46:12.612383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.612303    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-jzffv"]
Oct 30 00:46:12.616564 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:46:12.616532    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b2a498_5dc3_4d75_9859_f064b91e522f.slice/crio-27352f538fcd13f84bdb2dcc7924f5cfe8ecaec87ed4b89465425ea924a259e6 WatchSource:0}: Error finding container 27352f538fcd13f84bdb2dcc7924f5cfe8ecaec87ed4b89465425ea924a259e6: Status 404 returned error can't find the container with id 27352f538fcd13f84bdb2dcc7924f5cfe8ecaec87ed4b89465425ea924a259e6
Oct 30 00:46:12.639187 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.639153    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:12.865839 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.865743    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:12.876798 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:12.876761    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv" event={"ID":"c7b2a498-5dc3-4d75-9859-f064b91e522f","Type":"ContainerStarted","Data":"27352f538fcd13f84bdb2dcc7924f5cfe8ecaec87ed4b89465425ea924a259e6"}
Oct 30 00:46:13.068450 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:13.068400    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:14.291099 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:14.291064    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:16.896914 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:16.896883    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv" event={"ID":"c7b2a498-5dc3-4d75-9859-f064b91e522f","Type":"ContainerStarted","Data":"98ec2484aa991d3a99f2c33832339c729cd65468edbcc5646909bb3b98d20b1b"}
Oct 30 00:46:16.897360 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:16.897028    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv"
Oct 30 00:46:16.940533 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:16.940482    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv" podStartSLOduration=2.505625747 podStartE2EDuration="5.940468986s" totalImagesPullingTime="3.434843239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:16.939454562 +0000 UTC m=+457.010778398" watchObservedRunningTime="2026-10-30 00:46:16.940468986 +0000 UTC m=+457.011792821"
Oct 30 00:46:18.155727 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:18.155688    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:18.271668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:18.271633    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:18.381141 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:46:18.381109    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:46:18.466604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:18.466564    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:18.619291 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:18.619260    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:18.748349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:18.748260    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:19.014959 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:19.014880    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:19.141871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:19.141831    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:25.249695 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:25.249656    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:25.389173 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:25.389143    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:26.380600 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:26.380561    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:27.903624 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:27.903593    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-jzffv"
Oct 30 00:46:30.068341 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:30.068299    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:30.195751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:30.195718    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:32.503824 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:32.503791    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:32.639187 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:32.639147    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:35.812742 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:35.812708    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:39.534427 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:39.534375    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:39.665053 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:39.665015    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:39.849715 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:39.849622    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:39.985544 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:39.985509    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:40.103716 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:40.103632    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:47.064602 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:47.064565    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:47.202876 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:47.202844    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:48.380536 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:46:48.380496    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:46:49.896345 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:49.896313    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:51.853077 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:51.852995    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:52.034621 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:52.034583    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:54.288865 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:54.288828    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:46:54.397094 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:46:54.397056    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:00.411152 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:00.411120    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:00.640960 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:00.640914    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:16.380558 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:16.380518    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:16.499934 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:16.499907    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:16.608865 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:16.608829    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:17.275459 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.275418    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:17.385211 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.385167    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:17.817548 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.817516    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:47:17.820854 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.820832    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:17.838546 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.838521    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:47:17.851348 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.851324    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fblw9\""
Oct 30 00:47:17.851348 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.851327    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 00:47:17.875559 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.875529    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:47:17.882676 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.882651    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755cz\" (UniqueName: \"kubernetes.io/projected/d6c4055c-973c-4e7f-80cf-2dae06fae287-kube-api-access-755cz\") pod \"limitador-limitador-7c9b87d574-mxzkv\" (UID: \"d6c4055c-973c-4e7f-80cf-2dae06fae287\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:17.882826 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.882704    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6c4055c-973c-4e7f-80cf-2dae06fae287-config-file\") pod \"limitador-limitador-7c9b87d574-mxzkv\" (UID: \"d6c4055c-973c-4e7f-80cf-2dae06fae287\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:17.983803 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.983766    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755cz\" (UniqueName: \"kubernetes.io/projected/d6c4055c-973c-4e7f-80cf-2dae06fae287-kube-api-access-755cz\") pod \"limitador-limitador-7c9b87d574-mxzkv\" (UID: \"d6c4055c-973c-4e7f-80cf-2dae06fae287\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:17.983975 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.983827    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6c4055c-973c-4e7f-80cf-2dae06fae287-config-file\") pod \"limitador-limitador-7c9b87d574-mxzkv\" (UID: \"d6c4055c-973c-4e7f-80cf-2dae06fae287\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:17.984454 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:17.984436    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6c4055c-973c-4e7f-80cf-2dae06fae287-config-file\") pod \"limitador-limitador-7c9b87d574-mxzkv\" (UID: \"d6c4055c-973c-4e7f-80cf-2dae06fae287\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:18.004665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:18.004628    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755cz\" (UniqueName: \"kubernetes.io/projected/d6c4055c-973c-4e7f-80cf-2dae06fae287-kube-api-access-755cz\") pod \"limitador-limitador-7c9b87d574-mxzkv\" (UID: \"d6c4055c-973c-4e7f-80cf-2dae06fae287\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:18.135023 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:18.134939    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:18.380089 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:47:18.380052    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:47:18.427417 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:18.427394    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:47:18.429714 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:47:18.429683    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c4055c_973c_4e7f_80cf_2dae06fae287.slice/crio-7d66040cf2722e93c71ebe3addaa9de833b14ec9179a48fcd2df06ce6183490a WatchSource:0}: Error finding container 7d66040cf2722e93c71ebe3addaa9de833b14ec9179a48fcd2df06ce6183490a: Status 404 returned error can't find the container with id 7d66040cf2722e93c71ebe3addaa9de833b14ec9179a48fcd2df06ce6183490a
Oct 30 00:47:19.151136 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:19.151097    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv" event={"ID":"d6c4055c-973c-4e7f-80cf-2dae06fae287","Type":"ContainerStarted","Data":"7d66040cf2722e93c71ebe3addaa9de833b14ec9179a48fcd2df06ce6183490a"}
Oct 30 00:47:20.156913 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:20.156880    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv" event={"ID":"d6c4055c-973c-4e7f-80cf-2dae06fae287","Type":"ContainerStarted","Data":"f0b4d481c4a16ac7e71a3229390fe3c5679bd9fb93e72a6b1c9e5be23d279867"}
Oct 30 00:47:20.157338 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:20.157015    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:20.201649 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:20.201595    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv" podStartSLOduration=2.417312835 podStartE2EDuration="3.201582172s" totalImagesPullingTime="784.269337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:20.200451723 +0000 UTC m=+520.271775558" watchObservedRunningTime="2026-10-30 00:47:20.201582172 +0000 UTC m=+520.272906004"
Oct 30 00:47:22.103503 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:22.103460    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:26.484660 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:26.484626    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:31.161741 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:31.161713    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"
Oct 30 00:47:34.745279 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:34.745221    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:34.874179 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:34.874147    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:35.509999 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:35.509965    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:47.874579 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:47.874542    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:48.129217 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:48.129127    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:48.381306 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:47:48.381211    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:47:58.471850 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:58.471816    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:58.586450 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:58.586392    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:58.735149 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:58.735073    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:47:58.859404 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:47:58.859363    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:02.481113 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:02.481082    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:02.624561 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:02.624522    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:05.698536 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:05.698478    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:06.884958 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:06.884927    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:18.381325 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:48:18.381279    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:48:26.591583 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:26.591497    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:30.373254 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:30.373195    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:34.535983 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:34.535949    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:34.646651 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:34.646613    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:35.832727 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:35.832682    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:35.944464 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:35.944430    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:36.061711 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:36.061683    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:36.172185 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:36.172102    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:48:37.950871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:37.950828    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:48:46.818722 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:46.818684    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"]
Oct 30 00:48:46.822552 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:46.822529    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:46.839649 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:46.839623    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"]
Oct 30 00:48:46.933947 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:46.933920    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37c0bec0-ca1a-4194-bf8e-7c530a8b1aea-cert\") pod \"llmisvc-controller-manager-6466c554d8-fnvkh\" (UID: \"37c0bec0-ca1a-4194-bf8e-7c530a8b1aea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:46.933947 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:46.933960    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sl2p\" (UniqueName: \"kubernetes.io/projected/37c0bec0-ca1a-4194-bf8e-7c530a8b1aea-kube-api-access-6sl2p\") pod \"llmisvc-controller-manager-6466c554d8-fnvkh\" (UID: \"37c0bec0-ca1a-4194-bf8e-7c530a8b1aea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:47.035294 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:47.035261    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sl2p\" (UniqueName: \"kubernetes.io/projected/37c0bec0-ca1a-4194-bf8e-7c530a8b1aea-kube-api-access-6sl2p\") pod \"llmisvc-controller-manager-6466c554d8-fnvkh\" (UID: \"37c0bec0-ca1a-4194-bf8e-7c530a8b1aea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:47.035469 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:47.035347    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37c0bec0-ca1a-4194-bf8e-7c530a8b1aea-cert\") pod \"llmisvc-controller-manager-6466c554d8-fnvkh\" (UID: \"37c0bec0-ca1a-4194-bf8e-7c530a8b1aea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:47.037851 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:47.037826    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37c0bec0-ca1a-4194-bf8e-7c530a8b1aea-cert\") pod \"llmisvc-controller-manager-6466c554d8-fnvkh\" (UID: \"37c0bec0-ca1a-4194-bf8e-7c530a8b1aea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:47.056678 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:47.056651    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sl2p\" (UniqueName: \"kubernetes.io/projected/37c0bec0-ca1a-4194-bf8e-7c530a8b1aea-kube-api-access-6sl2p\") pod \"llmisvc-controller-manager-6466c554d8-fnvkh\" (UID: \"37c0bec0-ca1a-4194-bf8e-7c530a8b1aea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:47.135359 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:47.135277    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:47.435737 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:47.435675    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"]
Oct 30 00:48:47.438541 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:48:47.438516    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c0bec0_ca1a_4194_bf8e_7c530a8b1aea.slice/crio-6f33cea620822399bf51fafa93553bf66400e980cba6b0e3fe8fa0223a4d50ef WatchSource:0}: Error finding container 6f33cea620822399bf51fafa93553bf66400e980cba6b0e3fe8fa0223a4d50ef: Status 404 returned error can't find the container with id 6f33cea620822399bf51fafa93553bf66400e980cba6b0e3fe8fa0223a4d50ef
Oct 30 00:48:47.439807 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:47.439792    4576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:48:47.508505 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:47.508470    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh" event={"ID":"37c0bec0-ca1a-4194-bf8e-7c530a8b1aea","Type":"ContainerStarted","Data":"6f33cea620822399bf51fafa93553bf66400e980cba6b0e3fe8fa0223a4d50ef"}
Oct 30 00:48:48.379851 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:48:48.379814    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:48:48.514137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:48.514108    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh" event={"ID":"37c0bec0-ca1a-4194-bf8e-7c530a8b1aea","Type":"ContainerStarted","Data":"d2a1402352b6fffd83bc4d40ed2e4adb9bc4f7e43490cd005cad602a59887225"}
Oct 30 00:48:48.514370 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:48.514261    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:48:48.557539 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:48:48.557494    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh" podStartSLOduration=2.298120359 podStartE2EDuration="2.557480255s" totalImagesPullingTime="259.359896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:48.556482088 +0000 UTC m=+608.627805923" watchObservedRunningTime="2026-10-30 00:48:48.557480255 +0000 UTC m=+608.628804089"
Oct 30 00:49:18.380279 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:49:18.380222    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:49:19.519548 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:19.519522    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fnvkh"
Oct 30 00:49:19.603561 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:19.603532    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"]
Oct 30 00:49:19.603738 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:19.603718    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh" podUID="1592b36f-bac4-4be4-ab7b-bbbe440f51da" containerName="manager" containerID="cri-o://baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954" gracePeriod=30
Oct 30 00:49:19.845085 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:19.845063    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:49:19.933832 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:19.933795    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1592b36f-bac4-4be4-ab7b-bbbe440f51da-cert\" (UniqueName: \"kubernetes.io/secret/1592b36f-bac4-4be4-ab7b-bbbe440f51da-cert\") pod \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\" (UID: \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\") "
Oct 30 00:49:19.933832 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:19.933836    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1592b36f-bac4-4be4-ab7b-bbbe440f51da-kube-api-access-bfj76\" (UniqueName: \"kubernetes.io/projected/1592b36f-bac4-4be4-ab7b-bbbe440f51da-kube-api-access-bfj76\") pod \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\" (UID: \"1592b36f-bac4-4be4-ab7b-bbbe440f51da\") "
Oct 30 00:49:19.936205 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:19.936171    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1592b36f-bac4-4be4-ab7b-bbbe440f51da-cert" pod "1592b36f-bac4-4be4-ab7b-bbbe440f51da" (UID: "1592b36f-bac4-4be4-ab7b-bbbe440f51da"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:19.936339 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:19.936205    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1592b36f-bac4-4be4-ab7b-bbbe440f51da-kube-api-access-bfj76" pod "1592b36f-bac4-4be4-ab7b-bbbe440f51da" (UID: "1592b36f-bac4-4be4-ab7b-bbbe440f51da"). InnerVolumeSpecName "kube-api-access-bfj76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:20.035298 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.035192    4576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1592b36f-bac4-4be4-ab7b-bbbe440f51da-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:49:20.035298 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.035221    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfj76\" (UniqueName: \"kubernetes.io/projected/1592b36f-bac4-4be4-ab7b-bbbe440f51da-kube-api-access-bfj76\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:49:20.643607 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.643573    4576 generic.go:356] "Generic (PLEG): container finished" podID="1592b36f-bac4-4be4-ab7b-bbbe440f51da" containerID="baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954" exitCode=0
Oct 30 00:49:20.644029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.643621    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh" event={"ID":"1592b36f-bac4-4be4-ab7b-bbbe440f51da","Type":"ContainerDied","Data":"baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954"}
Oct 30 00:49:20.644029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.643624    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"
Oct 30 00:49:20.644029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.643657    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh" event={"ID":"1592b36f-bac4-4be4-ab7b-bbbe440f51da","Type":"ContainerDied","Data":"22e38e02d02c7f5def5f85aabf204babc31dee99e52f52c1982a1d6fa3c599f5"}
Oct 30 00:49:20.644029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.643678    4576 scope.go:122] "RemoveContainer" containerID="baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954"
Oct 30 00:49:20.651658 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.651641    4576 scope.go:122] "RemoveContainer" containerID="baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954"
Oct 30 00:49:20.651925 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:49:20.651909    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954\": container with ID starting with baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954 not found: ID does not exist" containerID="baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954"
Oct 30 00:49:20.651975 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.651935    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954"} err="failed to get container status \"baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954\": rpc error: code = NotFound desc = could not find container \"baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954\": container with ID starting with baeb08329f9499ce9d27d5cd5c3c3fbdeadefff3db62edf8308fca237bb9a954 not found: ID does not exist"
Oct 30 00:49:20.700382 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.700358    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"]
Oct 30 00:49:20.714366 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:20.714336    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xz2hh"]
Oct 30 00:49:22.105804 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:22.105772    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1592b36f-bac4-4be4-ab7b-bbbe440f51da" path="/var/lib/kubelet/pods/1592b36f-bac4-4be4-ab7b-bbbe440f51da/volumes"
Oct 30 00:49:22.417505 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:22.417422    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:22.551384 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:22.551346    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:26.721149 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:26.721108    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:40.214780 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:40.214741    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:41.290037 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:41.290001    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:49:42.385938 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:42.385897    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:43.685717 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:43.685678    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:49:43.918192 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:43.918154    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:45.615465 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:45.615427    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:46.793491 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:46.793447    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:48.379980 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:49:48.379946    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:49:49.497447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:49.497413    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:49.517563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:49.517525    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:49:50.935754 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:50.935715    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:49:56.088688 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:49:56.088594    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:50:10.987361 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:10.987322    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:11.163809 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:11.163777    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:16.122890 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.122857    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:16.165824 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.165786    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"]
Oct 30 00:50:16.166207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.166192    4576 cpu_manager.go:414] "containerMap: removing container" podUID="1592b36f-bac4-4be4-ab7b-bbbe440f51da" containerName="manager"
Oct 30 00:50:16.166280 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.166210    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1592b36f-bac4-4be4-ab7b-bbbe440f51da" containerName="manager"
Oct 30 00:50:16.166280 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.166276    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="1592b36f-bac4-4be4-ab7b-bbbe440f51da" containerName="manager"
Oct 30 00:50:16.169531 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.169516    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.198836 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.198804    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"]
Oct 30 00:50:16.200791 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.200765    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.200940 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.200800    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d434c59a-04fd-493c-a603-88326e67ae06-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.200940 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.200836    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.201304 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.201283    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.201416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.201317    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d434c59a-04fd-493c-a603-88326e67ae06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.201416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.201351    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.201535 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.201456    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649cg\" (UniqueName: \"kubernetes.io/projected/d434c59a-04fd-493c-a603-88326e67ae06-kube-api-access-649cg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.201589 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.201576    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.201643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.201608    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d434c59a-04fd-493c-a603-88326e67ae06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.229084 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.229058    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rxg2r\""
Oct 30 00:50:16.271574 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.271545    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:50:16.271574 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.271555    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:50:16.271779 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.271654    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:50:16.289767 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.289711    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:16.302863 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.302837    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649cg\" (UniqueName: \"kubernetes.io/projected/d434c59a-04fd-493c-a603-88326e67ae06-kube-api-access-649cg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.302996 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.302874    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.302996 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.302894    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d434c59a-04fd-493c-a603-88326e67ae06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303080 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303011    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303080 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303060    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d434c59a-04fd-493c-a603-88326e67ae06-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303162 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303141    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303300 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303281    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303300 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303290    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303412 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303319    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d434c59a-04fd-493c-a603-88326e67ae06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303412 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303340    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303412 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303349    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303644 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303625    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303782 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303701    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.303991 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.303856    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d434c59a-04fd-493c-a603-88326e67ae06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.305550 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.305515    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d434c59a-04fd-493c-a603-88326e67ae06-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.305815 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.305775    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d434c59a-04fd-493c-a603-88326e67ae06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.324881 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.324853    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d434c59a-04fd-493c-a603-88326e67ae06-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.325223 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.325189    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-649cg\" (UniqueName: \"kubernetes.io/projected/d434c59a-04fd-493c-a603-88326e67ae06-kube-api-access-649cg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8j72\" (UID: \"d434c59a-04fd-493c-a603-88326e67ae06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.483727 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.483695    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:16.798934 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.798850    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"]
Oct 30 00:50:16.802123 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:50:16.802094    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd434c59a_04fd_493c_a603_88326e67ae06.slice/crio-18a23dc430cd121796093bab16290e821bf075938d24a6f8f79bba5e281e23ae WatchSource:0}: Error finding container 18a23dc430cd121796093bab16290e821bf075938d24a6f8f79bba5e281e23ae: Status 404 returned error can't find the container with id 18a23dc430cd121796093bab16290e821bf075938d24a6f8f79bba5e281e23ae
Oct 30 00:50:16.870853 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:16.870815    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" event={"ID":"d434c59a-04fd-493c-a603-88326e67ae06","Type":"ContainerStarted","Data":"18a23dc430cd121796093bab16290e821bf075938d24a6f8f79bba5e281e23ae"}
Oct 30 00:50:18.383069 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:50:18.383036    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:50:20.278800 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:20.278769    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:50:20.279100 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:20.278837    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:50:20.279100 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:20.278867    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:50:20.894322 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:20.894283    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" event={"ID":"d434c59a-04fd-493c-a603-88326e67ae06","Type":"ContainerStarted","Data":"2ebfc13a84ae528931c96a8e692e194f69a476ccc94344928d4cf1fb46d06adb"}
Oct 30 00:50:20.970351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:20.970290    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" podStartSLOduration=1.496262623 podStartE2EDuration="4.970268255s" totalImagesPullingTime="3.474005632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:20.955115009 +0000 UTC m=+701.026438844" watchObservedRunningTime="2026-10-30 00:50:20.970268255 +0000 UTC m=+701.041592091"
Oct 30 00:50:21.483844 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:21.483809    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:21.486489 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:21.486442    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8j72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 00:50:21.486722 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:21.486697    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" podUID="d434c59a-04fd-493c-a603-88326e67ae06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 00:50:22.492399 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:22.492356    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8j72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 00:50:22.492892 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:22.492428    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" podUID="d434c59a-04fd-493c-a603-88326e67ae06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 00:50:23.487258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:23.485658    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8j72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 00:50:23.487258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:23.485721    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" podUID="d434c59a-04fd-493c-a603-88326e67ae06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 00:50:24.488259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:24.487666    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8j72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 00:50:24.488259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:24.487731    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" podUID="d434c59a-04fd-493c-a603-88326e67ae06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 00:50:25.484637 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:25.484601    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8j72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 00:50:25.484964 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:25.484933    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" podUID="d434c59a-04fd-493c-a603-88326e67ae06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 00:50:26.484966 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:26.484927    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8j72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 00:50:26.485394 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:26.484977    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" podUID="d434c59a-04fd-493c-a603-88326e67ae06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 00:50:26.841304 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:26.841201    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:27.484966 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:27.484930    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8j72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 00:50:27.485151 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:27.484997    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72" podUID="d434c59a-04fd-493c-a603-88326e67ae06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 00:50:28.487984 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:28.487956    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:28.488355 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:28.488208    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:28.488912 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:28.488896    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8j72"
Oct 30 00:50:28.910289 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:28.910160    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:31.208197 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.208155    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"]
Oct 30 00:50:31.211861 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.211841    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.243544 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.243514    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"]
Oct 30 00:50:31.367263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367199    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.367434 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367331    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.367434 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367362    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.367434 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367423    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.367558 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367466    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.367558 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367497    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.367558 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367520    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.367667 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367608    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee221b6-8c75-451b-b949-e90edd2185a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.367667 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.367638    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49gv\" (UniqueName: \"kubernetes.io/projected/4ee221b6-8c75-451b-b949-e90edd2185a6-kube-api-access-p49gv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468352    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49gv\" (UniqueName: \"kubernetes.io/projected/4ee221b6-8c75-451b-b949-e90edd2185a6-kube-api-access-p49gv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468395    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468426    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468454    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468494    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468526    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468564    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468590    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.468707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468636    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee221b6-8c75-451b-b949-e90edd2185a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.469005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.468945    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.469051 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.469011    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.469094 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.469069    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.469186 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.469126    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.469411 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.469392    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee221b6-8c75-451b-b949-e90edd2185a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.471564 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.471542    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.471860 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.471839    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.490211 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.490184    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee221b6-8c75-451b-b949-e90edd2185a6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.490427 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.490407    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49gv\" (UniqueName: \"kubernetes.io/projected/4ee221b6-8c75-451b-b949-e90edd2185a6-kube-api-access-p49gv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mb4zz\" (UID: \"4ee221b6-8c75-451b-b949-e90edd2185a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.525519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.525495    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:31.838127 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.835053    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"]
Oct 30 00:50:31.839359 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:50:31.839328    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee221b6_8c75_451b_b949_e90edd2185a6.slice/crio-198e424ff68eb3a6423bfaa8edfd54144e0bb5e3f7352822a3de70a7d575baaf WatchSource:0}: Error finding container 198e424ff68eb3a6423bfaa8edfd54144e0bb5e3f7352822a3de70a7d575baaf: Status 404 returned error can't find the container with id 198e424ff68eb3a6423bfaa8edfd54144e0bb5e3f7352822a3de70a7d575baaf
Oct 30 00:50:31.842276 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.842215    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:50:31.842369 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.842312    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:50:31.842408 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.842367    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:50:31.944174 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.944136    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" event={"ID":"4ee221b6-8c75-451b-b949-e90edd2185a6","Type":"ContainerStarted","Data":"6632e466b59060a72483d2ed97906fba29b44c7c325987220df625ecb5326b3e"}
Oct 30 00:50:31.944365 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.944193    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" event={"ID":"4ee221b6-8c75-451b-b949-e90edd2185a6","Type":"ContainerStarted","Data":"198e424ff68eb3a6423bfaa8edfd54144e0bb5e3f7352822a3de70a7d575baaf"}
Oct 30 00:50:31.992551 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:31.992502    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podStartSLOduration=0.992484048 podStartE2EDuration="992.484048ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:31.99183415 +0000 UTC m=+712.063157997" watchObservedRunningTime="2026-10-30 00:50:31.992484048 +0000 UTC m=+712.063807884"
Oct 30 00:50:32.526615 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:32.526581    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:32.527785 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:32.527765    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb4zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 00:50:32.527861 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:32.527809    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podUID="4ee221b6-8c75-451b-b949-e90edd2185a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 00:50:33.526208 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:33.526172    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb4zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 00:50:33.526396 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:33.526227    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podUID="4ee221b6-8c75-451b-b949-e90edd2185a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 00:50:34.526018 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:34.525985    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb4zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 00:50:34.526440 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:34.526039    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podUID="4ee221b6-8c75-451b-b949-e90edd2185a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 00:50:35.526354 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:35.526308    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb4zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 00:50:35.526763 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:35.526364    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podUID="4ee221b6-8c75-451b-b949-e90edd2185a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 00:50:36.526256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:36.526198    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb4zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 00:50:36.526446 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:36.526292    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podUID="4ee221b6-8c75-451b-b949-e90edd2185a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 00:50:37.526149 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:37.526104    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb4zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 00:50:37.526330 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:37.526163    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podUID="4ee221b6-8c75-451b-b949-e90edd2185a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 00:50:38.526837 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:38.526798    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb4zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 00:50:38.527264 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:38.526853    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podUID="4ee221b6-8c75-451b-b949-e90edd2185a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 00:50:39.936129 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:39.936093    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:40.020855 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:40.020824    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:40.021294 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:40.021270    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:40.022041 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:40.022025    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz"
Oct 30 00:50:40.426078 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:40.426040    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:41.974347 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:41.974311    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:42.111614 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:42.111571    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:43.412765 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:43.412725    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:43.574820 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:43.574778    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:50:48.383019 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:50:48.382972    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:50:55.597353 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:50:55.597318    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:18.380935 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:51:18.380898    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:51:26.943281 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:26.943182    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:29.019777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:29.019736    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:29.208543 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:29.208508    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:29.326442 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:29.326362    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:29.726665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:29.726621    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:29.858870 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:29.858835    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:30.094383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:30.094294    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:30.246591 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:30.246551    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:30.391076 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:30.390989    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:30.546843 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:30.546810    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:30.687044 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:30.687010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:30.832362 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:30.832327    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:33.426490 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:33.426403    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:33.571151 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:33.571113    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:33.729827 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:33.729797    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:34.307875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:34.307842    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:34.449969 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:34.449939    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:34.710079 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:34.710044    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:34.922719 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:34.922679    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:35.072974 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:35.072879    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:35.216841 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:35.216807    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:35.824787 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:35.824749    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:36.011651 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:36.011615    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:36.203541 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:36.203500    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:37.731158 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:37.731118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:37.859796 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:37.859767    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:38.651297 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:38.651213    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:38.772368 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:38.772316    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:39.611605 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:39.611573    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:39.754320 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:39.754285    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:40.577061 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:40.577027    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:40.695177 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:40.695142    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:40.810343 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:40.810312    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:42.051984 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:42.051952    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:42.280018 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:42.279984    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:42.319677 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:42.319599    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:51:42.491105 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:42.491067    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:42.618405 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:42.618317    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:43.124155 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:43.124109    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:43.261212 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:43.261174    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:44.467483 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:44.467449    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:44.587349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:44.587321    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:45.605343 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.605286    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:45.728639 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.728610    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:45.744988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.744955    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"]
Oct 30 00:51:45.748403 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.748386    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:45.769088 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.769056    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"]
Oct 30 00:51:45.781102 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.781072    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-298ddc\""
Oct 30 00:51:45.781102 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.781073    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-298ddc-dockercfg-jxsvv\""
Oct 30 00:51:45.819804 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.819769    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qbq\" (UniqueName: \"kubernetes.io/projected/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-kube-api-access-l6qbq\") pod \"payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n\" (UID: \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:45.819995 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.819908    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n\" (UID: \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:45.862460 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.862384    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"]
Oct 30 00:51:45.865673 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.865659    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:45.884327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.884296    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"]
Oct 30 00:51:45.921050 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.921016    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n\" (UID: \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:45.921219 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.921067    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4rsr\" (UniqueName: \"kubernetes.io/projected/7b1bc42f-9481-44bd-91a0-53f9454889c5-kube-api-access-q4rsr\") pod \"payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst\" (UID: \"7b1bc42f-9481-44bd-91a0-53f9454889c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:45.921219 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.921099    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qbq\" (UniqueName: \"kubernetes.io/projected/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-kube-api-access-l6qbq\") pod \"payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n\" (UID: \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:45.921219 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.921152    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1bc42f-9481-44bd-91a0-53f9454889c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst\" (UID: \"7b1bc42f-9481-44bd-91a0-53f9454889c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:45.921760 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.921739    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n\" (UID: \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:45.942902 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:45.942862    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qbq\" (UniqueName: \"kubernetes.io/projected/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-kube-api-access-l6qbq\") pod \"payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n\" (UID: \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:46.022551 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.022513    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4rsr\" (UniqueName: \"kubernetes.io/projected/7b1bc42f-9481-44bd-91a0-53f9454889c5-kube-api-access-q4rsr\") pod \"payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst\" (UID: \"7b1bc42f-9481-44bd-91a0-53f9454889c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:46.022759 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.022577    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1bc42f-9481-44bd-91a0-53f9454889c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst\" (UID: \"7b1bc42f-9481-44bd-91a0-53f9454889c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:46.023257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.023210    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1bc42f-9481-44bd-91a0-53f9454889c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst\" (UID: \"7b1bc42f-9481-44bd-91a0-53f9454889c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:46.044348 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.044323    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4rsr\" (UniqueName: \"kubernetes.io/projected/7b1bc42f-9481-44bd-91a0-53f9454889c5-kube-api-access-q4rsr\") pod \"payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst\" (UID: \"7b1bc42f-9481-44bd-91a0-53f9454889c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:46.061166 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.061146    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:46.178569 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.178541    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:46.355889 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.355779    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"]
Oct 30 00:51:46.358689 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:51:46.358665    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd86ca40_96ee_41dc_ae9f_1cf43cc7bb93.slice/crio-a335ed163661ef26b32b29c5e4ce00ff32cc6bcc27e76cf850b6b6027a3d6ee4 WatchSource:0}: Error finding container a335ed163661ef26b32b29c5e4ce00ff32cc6bcc27e76cf850b6b6027a3d6ee4: Status 404 returned error can't find the container with id a335ed163661ef26b32b29c5e4ce00ff32cc6bcc27e76cf850b6b6027a3d6ee4
Oct 30 00:51:46.470796 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.470752    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"]
Oct 30 00:51:46.473515 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:51:46.473487    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1bc42f_9481_44bd_91a0_53f9454889c5.slice/crio-15ec7de214d50c4da5de40f14a9e1948eae9fc0541d9281b4368f2b502c27340 WatchSource:0}: Error finding container 15ec7de214d50c4da5de40f14a9e1948eae9fc0541d9281b4368f2b502c27340: Status 404 returned error can't find the container with id 15ec7de214d50c4da5de40f14a9e1948eae9fc0541d9281b4368f2b502c27340
Oct 30 00:51:46.628206 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:46.628174    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:47.259898 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:47.259862    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n" event={"ID":"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93","Type":"ContainerStarted","Data":"a335ed163661ef26b32b29c5e4ce00ff32cc6bcc27e76cf850b6b6027a3d6ee4"}
Oct 30 00:51:47.261747 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:47.261680    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst" event={"ID":"7b1bc42f-9481-44bd-91a0-53f9454889c5","Type":"ContainerStarted","Data":"15ec7de214d50c4da5de40f14a9e1948eae9fc0541d9281b4368f2b502c27340"}
Oct 30 00:51:48.346128 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:48.346084    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:48.386996 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:51:48.386955    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:51:48.481187 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:48.481138    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:48.760307 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:48.760281    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:49.280472 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:49.280437    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n" event={"ID":"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93","Type":"ContainerStarted","Data":"9adb3e81f02a1516725e97bd04cea725887c5d34ea367a52ae677d136b8aa4fc"}
Oct 30 00:51:49.280677 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:49.280556    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:51:49.282140 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:49.282117    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst" event={"ID":"7b1bc42f-9481-44bd-91a0-53f9454889c5","Type":"ContainerStarted","Data":"e7fdb30aaf9d394afbb58facfe2c240789f51d180f3784ad8b9389bf0f84ba18"}
Oct 30 00:51:49.282282 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:49.282173    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:51:49.324174 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:49.324118    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n" podStartSLOduration=2.07575012 podStartE2EDuration="4.324099155s" totalImagesPullingTime="2.248349035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:49.322664448 +0000 UTC m=+789.393988317" watchObservedRunningTime="2026-10-30 00:51:49.324099155 +0000 UTC m=+789.395422991"
Oct 30 00:51:49.365285 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:49.365222    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst" podStartSLOduration=2.149046709 podStartE2EDuration="4.365208981s" totalImagesPullingTime="2.216162272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:49.364529488 +0000 UTC m=+789.435853316" watchObservedRunningTime="2026-10-30 00:51:49.365208981 +0000 UTC m=+789.436532865"
Oct 30 00:51:50.672856 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.672824    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"]
Oct 30 00:51:50.676467 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.676451    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.714413 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.714385    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:51:50.714623 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.714517    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 00:51:50.714723 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.714563    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:51:50.714723 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.714574    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-5a344d-serving-cert\""
Oct 30 00:51:50.714723 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.714705    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6w4xf\""
Oct 30 00:51:50.716432 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.716408    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"]
Oct 30 00:51:50.769477 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.769448    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-metrics-certs\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.769655 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.769511    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.769655 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.769619    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.769763 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.769653    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzphd\" (UniqueName: \"kubernetes.io/projected/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-kube-api-access-tzphd\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.801488 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.801452    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:50.870419 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.870383    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.870607 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.870434    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.870607 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.870462    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzphd\" (UniqueName: \"kubernetes.io/projected/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-kube-api-access-tzphd\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.870607 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.870522    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-metrics-certs\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.871000 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.870977    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.873032 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.873010    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.873140 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.873122    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-metrics-certs\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.914817 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.914785    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzphd\" (UniqueName: \"kubernetes.io/projected/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-kube-api-access-tzphd\") pod \"maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:50.937927 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.937895    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:50.989082 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:50.989053    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:51.058135 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:51.058084    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:51.285092 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:51.285069    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"]
Oct 30 00:51:51.287718 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:51:51.287689    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aa3b15_f1c8_4b72_951e_c215c8c4e7fe.slice/crio-3a78c9299878e1bcdb347a506f5f80728731bde35fc4b6e3af619ffba6960996 WatchSource:0}: Error finding container 3a78c9299878e1bcdb347a506f5f80728731bde35fc4b6e3af619ffba6960996: Status 404 returned error can't find the container with id 3a78c9299878e1bcdb347a506f5f80728731bde35fc4b6e3af619ffba6960996
Oct 30 00:51:52.025412 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:52.025375    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:51:52.297458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:52.297371    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p" event={"ID":"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe","Type":"ContainerStarted","Data":"3a78c9299878e1bcdb347a506f5f80728731bde35fc4b6e3af619ffba6960996"}
Oct 30 00:51:53.905303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:53.905275    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:54.306527 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:54.306490    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p" event={"ID":"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe","Type":"ContainerStarted","Data":"c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc"}
Oct 30 00:51:54.306736 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:54.306615    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:51:54.349393 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:54.349342    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p" podStartSLOduration=1.875069466 podStartE2EDuration="4.349328733s" totalImagesPullingTime="2.474259267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:54.348712866 +0000 UTC m=+794.420036703" watchObservedRunningTime="2026-10-30 00:51:54.349328733 +0000 UTC m=+794.420652570"
Oct 30 00:51:56.205822 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:56.205778    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:56.332498 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:56.332460    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:56.454809 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:56.454775    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:58.372204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:58.372163    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:58.743713 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:58.743679    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:58.964526 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:58.964497    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:59.289735 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:59.289694    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:51:59.348217 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:59.348179    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:51:59.473443 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:51:59.473407    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:00.289960 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:00.289936    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 00:52:00.290157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:00.289996    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 00:52:00.316522 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:00.316497    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 00:52:01.467983 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:01.467949    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:02.292576 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.292542    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"]
Oct 30 00:52:02.296520 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.296469    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.321489 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.321462    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"]
Oct 30 00:52:02.330698 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.330673    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8cf33a\"/\"kube-root-ca.crt\""
Oct 30 00:52:02.330827 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.330702    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8cf33a\"/\"openshift-service-ca.crt\""
Oct 30 00:52:02.330827 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.330723    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8cf33a\"/\"default-dockercfg-t4sjj\""
Oct 30 00:52:02.333640 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.333621    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8cf33a\"/\"simulator-w1-8cf33a-kserve-self-signed-certs\""
Oct 30 00:52:02.386874 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.386840    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:02.478687 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.478652    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7k64\" (UniqueName: \"kubernetes.io/projected/2bed9833-4f1e-463d-a660-04bace8eb95b-kube-api-access-m7k64\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.479098 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.478697    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-home\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.479098 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.478714    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-dshm\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.479098 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.478809    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-tmp-dir\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.479098 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.478949    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-model-cache\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.479098 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.478967    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bed9833-4f1e-463d-a660-04bace8eb95b-tls-certs\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.579981 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.579880    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-model-cache\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.579981 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.579932    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bed9833-4f1e-463d-a660-04bace8eb95b-tls-certs\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.580173 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.579981    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7k64\" (UniqueName: \"kubernetes.io/projected/2bed9833-4f1e-463d-a660-04bace8eb95b-kube-api-access-m7k64\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.580173 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.580133    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-home\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.580299 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.580172    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-dshm\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.580299 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.580214    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-tmp-dir\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.580403 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.580353    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-model-cache\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.580490 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.580471    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-home\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.580587 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.580564    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-tmp-dir\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.582497 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.582475    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-dshm\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.582742 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.582722    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bed9833-4f1e-463d-a660-04bace8eb95b-tls-certs\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.601183 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.601161    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7k64\" (UniqueName: \"kubernetes.io/projected/2bed9833-4f1e-463d-a660-04bace8eb95b-kube-api-access-m7k64\") pod \"simulator-w1-8cf33a-kserve-68998b644b-9c6bt\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") " pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.618076 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.618043    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:02.922877 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:02.922854    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"]
Oct 30 00:52:02.925326 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:52:02.925295    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bed9833_4f1e_463d_a660_04bace8eb95b.slice/crio-f54b24f6f3e73f28cbfd17966ff11f4f38dc1b58867709adedc470b30ab712c8 WatchSource:0}: Error finding container f54b24f6f3e73f28cbfd17966ff11f4f38dc1b58867709adedc470b30ab712c8: Status 404 returned error can't find the container with id f54b24f6f3e73f28cbfd17966ff11f4f38dc1b58867709adedc470b30ab712c8
Oct 30 00:52:03.347031 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:03.346993    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" event={"ID":"2bed9833-4f1e-463d-a660-04bace8eb95b","Type":"ContainerStarted","Data":"f54b24f6f3e73f28cbfd17966ff11f4f38dc1b58867709adedc470b30ab712c8"}
Oct 30 00:52:03.524307 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:03.524194    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:03.683516 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:03.683425    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:04.130097 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:04.130064    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:04.307057 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:04.306958    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:04.353744 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:04.353701    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" event={"ID":"2bed9833-4f1e-463d-a660-04bace8eb95b","Type":"ContainerStarted","Data":"d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6"}
Oct 30 00:52:04.406471 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:04.406410    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podStartSLOduration=1.294427896 podStartE2EDuration="2.406397011s" totalImagesPullingTime="1.111969115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:04.404520318 +0000 UTC m=+804.475844153" watchObservedRunningTime="2026-10-30 00:52:04.406397011 +0000 UTC m=+804.477720879"
Oct 30 00:52:04.528824 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:04.528750    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:04.618813 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:04.618735    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:05.158011 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:05.157974    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:05.360794 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:05.360761    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:05.633283 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:05.633219    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:07.530400 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:07.530360    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:09.643146 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:09.643112    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:09.894332 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:09.894206    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:10.213085 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:10.213050    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:10.335843 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:10.335790    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:10.462415 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:10.462379    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:12.619155 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.619119    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:12.632461 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.632436    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:12.760664 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.760627    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"]
Oct 30 00:52:12.764926 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.764898    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.775021 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.774993    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c650a359-b53a-4583-ae54-c634952769a4-tls-certs\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.775137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.775058    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-model-cache\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.775199 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.775166    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-home\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.775270 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.775206    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-tmp-dir\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.775492 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.775367    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-dshm\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.775492 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.775445    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhq5\" (UniqueName: \"kubernetes.io/projected/c650a359-b53a-4583-ae54-c634952769a4-kube-api-access-ldhq5\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.781221 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.781196    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7d2b75\"/\"kube-root-ca.crt\""
Oct 30 00:52:12.781399 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.781375    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"]
Oct 30 00:52:12.781471 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.781381    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7d2b75\"/\"simulator-w2-7d2b75-kserve-self-signed-certs\""
Oct 30 00:52:12.781471 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.781427    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7d2b75\"/\"openshift-service-ca.crt\""
Oct 30 00:52:12.781471 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.781435    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7d2b75\"/\"default-dockercfg-8gkch\""
Oct 30 00:52:12.839190 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.839148    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:12.876503 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.876429    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-model-cache\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.876503 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.876487    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-home\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.876732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.876510    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-tmp-dir\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.876732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.876549    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-dshm\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.876732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.876572    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhq5\" (UniqueName: \"kubernetes.io/projected/c650a359-b53a-4583-ae54-c634952769a4-kube-api-access-ldhq5\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.876732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.876615    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c650a359-b53a-4583-ae54-c634952769a4-tls-certs\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.876967 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.876946    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-model-cache\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.877021 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.876963    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-home\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.877021 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.877010    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-tmp-dir\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.879275 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.879224    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-dshm\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.879420 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.879401    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c650a359-b53a-4583-ae54-c634952769a4-tls-certs\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.897204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.897180    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhq5\" (UniqueName: \"kubernetes.io/projected/c650a359-b53a-4583-ae54-c634952769a4-kube-api-access-ldhq5\") pod \"simulator-w2-7d2b75-kserve-66c999c468-dsg2d\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") " pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:12.971871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:12.971839    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:52:13.078160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:13.078118    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:13.370636 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:13.370603    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"]
Oct 30 00:52:13.374074 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:52:13.374046    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc650a359_b53a_4583_ae54_c634952769a4.slice/crio-31dd9b061b3b32ff2eb074e42e7d4c8689f0c8862427b31af2ae37ca33784166 WatchSource:0}: Error finding container 31dd9b061b3b32ff2eb074e42e7d4c8689f0c8862427b31af2ae37ca33784166: Status 404 returned error can't find the container with id 31dd9b061b3b32ff2eb074e42e7d4c8689f0c8862427b31af2ae37ca33784166
Oct 30 00:52:13.396872 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:13.396842    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" event={"ID":"c650a359-b53a-4583-ae54-c634952769a4","Type":"ContainerStarted","Data":"31dd9b061b3b32ff2eb074e42e7d4c8689f0c8862427b31af2ae37ca33784166"}
Oct 30 00:52:13.408866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:13.408844    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:52:13.664181 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:13.664099    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:14.338458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.338424    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"]
Oct 30 00:52:14.352423 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.352390    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.360422 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.360393    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"]
Oct 30 00:52:14.371410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.371377    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"openshift-service-ca.crt\""
Oct 30 00:52:14.371410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.371387    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"kube-root-ca.crt\""
Oct 30 00:52:14.371570 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.371523    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"default-dockercfg-jgw45\""
Oct 30 00:52:14.382424 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.382404    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3c2596\"/\"simulator-w0-3c2596-kserve-self-signed-certs\""
Oct 30 00:52:14.389819 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.389791    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-home\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.389954 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.389862    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-model-cache\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.389954 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.389936    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea057e-546a-43b4-b257-74dd27f07b08-tls-certs\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.390055 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.390028    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-dshm\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.390099 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.390069    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7dw\" (UniqueName: \"kubernetes.io/projected/b8ea057e-546a-43b4-b257-74dd27f07b08-kube-api-access-mv7dw\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.390193 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.390175    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-tmp-dir\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.403598 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.403558    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" event={"ID":"c650a359-b53a-4583-ae54-c634952769a4","Type":"ContainerStarted","Data":"f6b89fc2e3be30ebdef7edded0c750269d43ab21f4a7ad4f553c1b1409af3a9d"}
Oct 30 00:52:14.408840 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.408808    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:14.445456 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.445404    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podStartSLOduration=2.445392965 podStartE2EDuration="2.445392965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:14.444896815 +0000 UTC m=+814.516220673" watchObservedRunningTime="2026-10-30 00:52:14.445392965 +0000 UTC m=+814.516716799"
Oct 30 00:52:14.490871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.490831    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-model-cache\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.491048 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.490888    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea057e-546a-43b4-b257-74dd27f07b08-tls-certs\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.491048 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.490933    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-dshm\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.491048 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.490955    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7dw\" (UniqueName: \"kubernetes.io/projected/b8ea057e-546a-43b4-b257-74dd27f07b08-kube-api-access-mv7dw\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.491048 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.491015    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-tmp-dir\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.491206 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.491067    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-home\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.491319 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.491299    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-model-cache\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.491414 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.491395    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-home\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.491842 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.491824    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-tmp-dir\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.493888 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.493856    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-dshm\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.494072 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.494053    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea057e-546a-43b4-b257-74dd27f07b08-tls-certs\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.512210 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.512178    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7dw\" (UniqueName: \"kubernetes.io/projected/b8ea057e-546a-43b4-b257-74dd27f07b08-kube-api-access-mv7dw\") pod \"simulator-w0-3c2596-kserve-9569fc9f5-dzf28\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") " pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.667385 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.667297    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:14.826251 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.826209    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:14.945195 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.945164    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:14.962068 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:14.961981    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"]
Oct 30 00:52:14.965631 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:52:14.965594    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8ea057e_546a_43b4_b257_74dd27f07b08.slice/crio-b32aac6623b669362d1fc2e9625d8329a056026088b110911d8505ddf846baf0 WatchSource:0}: Error finding container b32aac6623b669362d1fc2e9625d8329a056026088b110911d8505ddf846baf0: Status 404 returned error can't find the container with id b32aac6623b669362d1fc2e9625d8329a056026088b110911d8505ddf846baf0
Oct 30 00:52:15.078296 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.078265    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:15.410205 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.410116    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" event={"ID":"b8ea057e-546a-43b4-b257-74dd27f07b08","Type":"ContainerStarted","Data":"bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720"}
Oct 30 00:52:15.410205 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.410158    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" event={"ID":"b8ea057e-546a-43b4-b257-74dd27f07b08","Type":"ContainerStarted","Data":"b32aac6623b669362d1fc2e9625d8329a056026088b110911d8505ddf846baf0"}
Oct 30 00:52:15.412605 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.412574    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:15.452409 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.452357    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podStartSLOduration=1.452344163 podStartE2EDuration="1.452344163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:15.451298683 +0000 UTC m=+815.522622517" watchObservedRunningTime="2026-10-30 00:52:15.452344163 +0000 UTC m=+815.523667997"
Oct 30 00:52:15.531916 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.531883    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:15.602306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.602267    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"]
Oct 30 00:52:15.606789 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.606766    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.630043 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.630014    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"]
Oct 30 00:52:15.633298 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.632727    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5a344d-openshift-default-dockercfg-7z8cg\""
Oct 30 00:52:15.667866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.667783    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:15.669274 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.669193    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:15.702778 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.702740    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.702778 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.702789    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.702976 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.702907    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vb4m\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-kube-api-access-9vb4m\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.702976 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.702968    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.703050 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.703027    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.703086 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.703071    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.703128 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.703092    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.703128 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.703117    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.703191 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.703150    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.803766 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.803731    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.803766 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.803770    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.803992 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.803894    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vb4m\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-kube-api-access-9vb4m\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.803992 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.803979    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804086 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804000    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804086 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804016    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804086 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804039    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804086 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804071    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804282 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804094    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804282 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804193    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804440 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804417    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804500 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804443    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804558 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804529    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.804615 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.804550    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.806405 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.806381    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.806813 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.806782    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.826575 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.826547    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.826682 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.826588    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vb4m\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-kube-api-access-9vb4m\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:15.921460 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:15.921380    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:16.241970 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.241925    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"]
Oct 30 00:52:16.244201 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:52:16.244171    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3879463c_e6a8_4a5c_92da_0ec47e7cf070.slice/crio-2b208ae80119e2d49ebeb149ffe893a29e3d378444f2349081189d2677c4b0cf WatchSource:0}: Error finding container 2b208ae80119e2d49ebeb149ffe893a29e3d378444f2349081189d2677c4b0cf: Status 404 returned error can't find the container with id 2b208ae80119e2d49ebeb149ffe893a29e3d378444f2349081189d2677c4b0cf
Oct 30 00:52:16.246901 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.246861    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:52:16.246975 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.246956    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:52:16.247019 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.247004    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:52:16.416651 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.416609    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:16.416844 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.416782    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" event={"ID":"3879463c-e6a8-4a5c-92da-0ec47e7cf070","Type":"ContainerStarted","Data":"ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9"}
Oct 30 00:52:16.416844 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.416829    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" event={"ID":"3879463c-e6a8-4a5c-92da-0ec47e7cf070","Type":"ContainerStarted","Data":"2b208ae80119e2d49ebeb149ffe893a29e3d378444f2349081189d2677c4b0cf"}
Oct 30 00:52:16.462612 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.462556    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" podStartSLOduration=1.4625375 podStartE2EDuration="1.4625375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:16.460918322 +0000 UTC m=+816.532242150" watchObservedRunningTime="2026-10-30 00:52:16.4625375 +0000 UTC m=+816.533861336"
Oct 30 00:52:16.921770 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:16.921734    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:17.923220 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:17.923185    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:52:17.923632 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:17.923263    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:52:18.182817 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.182721    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"]
Oct 30 00:52:18.208081 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.207453    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"]
Oct 30 00:52:18.208081 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.207577    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.224227 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.224203    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"simulator-w5-da76a2-kserve-self-signed-certs\""
Oct 30 00:52:18.226810 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.226781    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-dshm\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.226981 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.226957    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-model-cache\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.227082 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.227065    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-home\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.227135 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.227115    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxmc\" (UniqueName: \"kubernetes.io/projected/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-kube-api-access-smxmc\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.227265 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.227221    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tmp-dir\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.227390 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.227299    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tls-certs\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.237802 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.237780    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"default-dockercfg-jtgwd\""
Oct 30 00:52:18.237920 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.237898    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"kube-root-ca.crt\""
Oct 30 00:52:18.237973 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.237939    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"openshift-service-ca.crt\""
Oct 30 00:52:18.259376 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.259337    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:18.328816 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.328789    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-model-cache\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.329004 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.328824    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-home\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.329004 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.328852    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxmc\" (UniqueName: \"kubernetes.io/projected/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-kube-api-access-smxmc\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.329004 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.328957    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tmp-dir\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.329004 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.328989    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tls-certs\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.329266 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.329024    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-dshm\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.329649 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.329620    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tmp-dir\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.329761 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.329646    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-model-cache\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.329761 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.329714    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-home\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.331606 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.331584    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-dshm\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.332370 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.332345    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tls-certs\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.351903 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.351878    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxmc\" (UniqueName: \"kubernetes.io/projected/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-kube-api-access-smxmc\") pod \"simulator-w5-da76a2-kserve-5ffd95d859-2vxpx\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") " pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.421070 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:52:18.421027    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:52:18.520718 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.520678    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:18.749719 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.749683    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:18.841047 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.841018    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"]
Oct 30 00:52:18.843649 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:52:18.843616    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53955fbc_3bd0_4d7a_bd60_25f05edbcb7b.slice/crio-55f225f3a979eb4ff4dec7d10b30b5200a4498e9cbb7f933b2d0a59a0304a2e0 WatchSource:0}: Error finding container 55f225f3a979eb4ff4dec7d10b30b5200a4498e9cbb7f933b2d0a59a0304a2e0: Status 404 returned error can't find the container with id 55f225f3a979eb4ff4dec7d10b30b5200a4498e9cbb7f933b2d0a59a0304a2e0
Oct 30 00:52:18.922338 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.922309    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:52:18.922453 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:18.922361    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:52:19.440277 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:19.440208    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" event={"ID":"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b","Type":"ContainerStarted","Data":"7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315"}
Oct 30 00:52:19.440277 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:19.440278    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" event={"ID":"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b","Type":"ContainerStarted","Data":"55f225f3a979eb4ff4dec7d10b30b5200a4498e9cbb7f933b2d0a59a0304a2e0"}
Oct 30 00:52:19.500301 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:19.491374    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podStartSLOduration=1.491353832 podStartE2EDuration="1.491353832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:19.490640289 +0000 UTC m=+819.561964125" watchObservedRunningTime="2026-10-30 00:52:19.491353832 +0000 UTC m=+819.562677668"
Oct 30 00:52:19.521620 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:19.521588    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:19.922474 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:19.922395    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:52:19.922474 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:19.922450    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:52:20.922647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:20.922581    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:52:20.923301 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:20.922657    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:52:21.007900 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:21.007868    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:21.115643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:21.115597    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:21.583257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:21.583206    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:21.702330 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:21.702295    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:21.829011 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:21.828975    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:21.921921 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:21.921846    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 00:52:21.921921 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:21.921907    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 00:52:22.808382 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:22.808353    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:22.925995 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:22.925964    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:22.926286 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:22.926263    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:22.927084 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:22.927068    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 00:52:23.078897 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:23.078801    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:23.091572 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:23.091547    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:23.470422 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:23.470397    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:52:24.575515 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:24.575470    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:24.667952 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:24.667921    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:24.680807 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:24.680785    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:25.481190 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:25.481159    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:52:26.174896 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:26.174859    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:27.071600 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:27.071566    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:27.172257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:27.172200    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:52:27.942788 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:27.942759    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:27.977226 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:27.977194    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"]
Oct 30 00:52:27.986263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:27.986206    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:27.998787 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:27.998758    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"]
Oct 30 00:52:28.010283 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.010208    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"openshift-service-ca.crt\""
Oct 30 00:52:28.010283 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.010206    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"kube-root-ca.crt\""
Oct 30 00:52:28.010283 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.010258    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"default-dockercfg-7hgn4\""
Oct 30 00:52:28.016269 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.016248    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b565ec\"/\"simulator-w4-b565ec-kserve-self-signed-certs\""
Oct 30 00:52:28.063203 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.063167    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:28.124523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.124483    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-model-cache\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.124523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.124532    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-home\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.124794 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.124631    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt6n4\" (UniqueName: \"kubernetes.io/projected/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-kube-api-access-lt6n4\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.124794 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.124734    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tls-certs\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.124794 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.124759    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tmp-dir\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.124794 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.124782    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-dshm\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226161 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226074    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tls-certs\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226161 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226112    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tmp-dir\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226161 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226131    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-dshm\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226429 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226190    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-model-cache\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226429 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226208    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-home\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226429 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226266    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt6n4\" (UniqueName: \"kubernetes.io/projected/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-kube-api-access-lt6n4\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226694 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226659    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-home\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226831 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226702    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-model-cache\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.226831 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.226777    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tmp-dir\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.228538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.228515    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-dshm\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.229107 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.229081    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tls-certs\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.247327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.247265    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt6n4\" (UniqueName: \"kubernetes.io/projected/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-kube-api-access-lt6n4\") pod \"simulator-w4-b565ec-kserve-d68c79f-w8czl\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") " pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.301810 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.301782    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:28.521578 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.521504    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:28.534185 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.534160    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:28.594578 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:28.594551    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"]
Oct 30 00:52:28.599563 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:52:28.599532    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded86ab45_a7e9_4335_b98a_8a98dd88f83e.slice/crio-c71da735fbaf2068804c5e8edd730e32f78aac9ba003254e7cd1a66f9816693d WatchSource:0}: Error finding container c71da735fbaf2068804c5e8edd730e32f78aac9ba003254e7cd1a66f9816693d: Status 404 returned error can't find the container with id c71da735fbaf2068804c5e8edd730e32f78aac9ba003254e7cd1a66f9816693d
Oct 30 00:52:29.149087 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:29.149055    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:29.487866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:29.487823    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" event={"ID":"ed86ab45-a7e9-4335-b98a-8a98dd88f83e","Type":"ContainerStarted","Data":"a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e"}
Oct 30 00:52:29.488044 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:29.487877    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" event={"ID":"ed86ab45-a7e9-4335-b98a-8a98dd88f83e","Type":"ContainerStarted","Data":"c71da735fbaf2068804c5e8edd730e32f78aac9ba003254e7cd1a66f9816693d"}
Oct 30 00:52:29.499030 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:29.499003    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:52:29.531210 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:29.531165    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podStartSLOduration=2.531151679 podStartE2EDuration="2.531151679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:29.52962183 +0000 UTC m=+829.600945665" watchObservedRunningTime="2026-10-30 00:52:29.531151679 +0000 UTC m=+829.602475515"
Oct 30 00:52:29.617535 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:29.617499    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:30.302392 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:30.302357    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:30.638668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:30.638549    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:30.747424 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:30.747393    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:31.262799 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:31.262750    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:31.415437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:31.415406    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:33.021154 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:33.021121    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:34.577364 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:34.577316    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:35.539702 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:35.539661    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:36.172864 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:36.172832    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:37.697563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:37.697520    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:38.302500 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:38.302466    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:38.321729 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:38.321701    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:38.541660 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:38.541628    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:52:38.696848 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:38.696821    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:52:39.498002 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.497965    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:39.686469 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.686434    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"]
Oct 30 00:52:39.693648 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.693619    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.712631 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.712604    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"]
Oct 30 00:52:39.724100 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.724073    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-298ddc\"/\"kube-root-ca.crt\""
Oct 30 00:52:39.724372 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.724332    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-298ddc\"/\"openshift-service-ca.crt\""
Oct 30 00:52:39.724563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.724377    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-298ddc\"/\"default-dockercfg-kbp2n\""
Oct 30 00:52:39.724563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.724506    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-298ddc\"/\"simulator-w3-298ddc-kserve-self-signed-certs\""
Oct 30 00:52:39.755971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.755895    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:39.842516 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.842474    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42wzq\" (UniqueName: \"kubernetes.io/projected/85b941c6-c18c-46bd-be7a-58b01a29b05f-kube-api-access-42wzq\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.842718 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.842546    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-tmp-dir\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.842718 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.842603    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-home\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.842718 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.842627    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-model-cache\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.842871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.842779    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b941c6-c18c-46bd-be7a-58b01a29b05f-tls-certs\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.842871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.842820    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-dshm\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944247 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944211    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b941c6-c18c-46bd-be7a-58b01a29b05f-tls-certs\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944430 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944270    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-dshm\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944430 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944323    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42wzq\" (UniqueName: \"kubernetes.io/projected/85b941c6-c18c-46bd-be7a-58b01a29b05f-kube-api-access-42wzq\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944430 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944382    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-tmp-dir\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944430 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944409    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-home\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944627 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944433    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-model-cache\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944876 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944846    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-model-cache\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944999 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944887    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-tmp-dir\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.944999 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.944973    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-home\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.946894 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.946858    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-dshm\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.947054 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.947032    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b941c6-c18c-46bd-be7a-58b01a29b05f-tls-certs\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:39.965121 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:39.965091    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42wzq\" (UniqueName: \"kubernetes.io/projected/85b941c6-c18c-46bd-be7a-58b01a29b05f-kube-api-access-42wzq\") pod \"simulator-w3-298ddc-kserve-589cbf9754-4kbgt\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") " pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:40.020295 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:40.020192    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:40.323527 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:40.323504    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"]
Oct 30 00:52:40.325850 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:52:40.325822    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85b941c6_c18c_46bd_be7a_58b01a29b05f.slice/crio-d203687d1be04cd34538b4fee301008802b58d37b6336325df2fda0ecfd4d724 WatchSource:0}: Error finding container d203687d1be04cd34538b4fee301008802b58d37b6336325df2fda0ecfd4d724: Status 404 returned error can't find the container with id d203687d1be04cd34538b4fee301008802b58d37b6336325df2fda0ecfd4d724
Oct 30 00:52:40.540985 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:40.540954    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" event={"ID":"85b941c6-c18c-46bd-be7a-58b01a29b05f","Type":"ContainerStarted","Data":"fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b"}
Oct 30 00:52:40.540985 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:40.540992    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" event={"ID":"85b941c6-c18c-46bd-be7a-58b01a29b05f","Type":"ContainerStarted","Data":"d203687d1be04cd34538b4fee301008802b58d37b6336325df2fda0ecfd4d724"}
Oct 30 00:52:40.582809 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:40.582722    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podStartSLOduration=1.582708418 podStartE2EDuration="1.582708418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:40.582570212 +0000 UTC m=+840.653894046" watchObservedRunningTime="2026-10-30 00:52:40.582708418 +0000 UTC m=+840.654032253"
Oct 30 00:52:41.020942 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:41.020917    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:41.154694 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:41.154662    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:42.610647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:42.610609    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:44.066156 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:44.066110    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:45.508358 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:45.508319    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:46.882669 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:46.882637    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"]
Oct 30 00:52:46.886716 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:46.886694    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:46.906593 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:46.906567    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"]
Oct 30 00:52:46.918766 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:46.918734    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7d2b75\"/\"premium-w2-7d2b75-kserve-self-signed-certs\""
Oct 30 00:52:46.947489 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:46.947461    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:47.012511 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.012476    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-model-cache\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.012685 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.012591    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d506eb44-fd4e-4c9a-a553-6369313404a7-tls-certs\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.012685 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.012660    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-dshm\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.012783 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.012753    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtdf6\" (UniqueName: \"kubernetes.io/projected/d506eb44-fd4e-4c9a-a553-6369313404a7-kube-api-access-vtdf6\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.012841 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.012791    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-tmp-dir\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.012841 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.012810    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-home\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.113779 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.113748    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-home\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.113966 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.113805    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-model-cache\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.113966 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.113875    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d506eb44-fd4e-4c9a-a553-6369313404a7-tls-certs\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.113966 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.113929    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-dshm\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.114171 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.113988    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtdf6\" (UniqueName: \"kubernetes.io/projected/d506eb44-fd4e-4c9a-a553-6369313404a7-kube-api-access-vtdf6\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.114171 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.114044    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-tmp-dir\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.114340 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.114314    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-home\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.114401 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.114326    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-model-cache\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.114873 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.114850    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-tmp-dir\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.116749 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.116726    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-dshm\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.116945 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.116926    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d506eb44-fd4e-4c9a-a553-6369313404a7-tls-certs\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.135636 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.135567    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtdf6\" (UniqueName: \"kubernetes.io/projected/d506eb44-fd4e-4c9a-a553-6369313404a7-kube-api-access-vtdf6\") pod \"premium-w2-7d2b75-kserve-5495bc6569-hzfmd\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") " pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.194446 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.194366    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:47.223440 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.223409    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:47.525047 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.525023    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"]
Oct 30 00:52:47.527782 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:52:47.527752    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd506eb44_fd4e_4c9a_a553_6369313404a7.slice/crio-4a9394f913a8df54f46d5290454fce1696251ca66e76fba5ed322749fb5ba2b8 WatchSource:0}: Error finding container 4a9394f913a8df54f46d5290454fce1696251ca66e76fba5ed322749fb5ba2b8: Status 404 returned error can't find the container with id 4a9394f913a8df54f46d5290454fce1696251ca66e76fba5ed322749fb5ba2b8
Oct 30 00:52:47.578288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:47.576939    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" event={"ID":"d506eb44-fd4e-4c9a-a553-6369313404a7","Type":"ContainerStarted","Data":"4a9394f913a8df54f46d5290454fce1696251ca66e76fba5ed322749fb5ba2b8"}
Oct 30 00:52:48.383088 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:52:48.383048    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:52:48.582967 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:48.582929    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" event={"ID":"d506eb44-fd4e-4c9a-a553-6369313404a7","Type":"ContainerStarted","Data":"606b8401af82033f47d12f7750ab511a574cd6f830b2ff09b5044d74d7b5c702"}
Oct 30 00:52:48.624588 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:48.624542    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podStartSLOduration=2.624531134 podStartE2EDuration="2.624531134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:48.623744238 +0000 UTC m=+848.695068075" watchObservedRunningTime="2026-10-30 00:52:48.624531134 +0000 UTC m=+848.695854968"
Oct 30 00:52:48.762508 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:48.762478    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:49.224029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:49.223998    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:50.020599 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:50.020564    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:50.033964 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:50.033940    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:50.290538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:50.290444    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:50.604220 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:50.604141    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:52:51.866418 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:51.866324    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:52.625911 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:52.625875    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:52.671715 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:52.671679    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:52:53.378410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:53.378371    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:55.055683 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:55.055638    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:56.576221 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:56.576188    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:57.223739 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:57.223708    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:57.237000 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:57.236971    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:57.639969 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:57.639884    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:52:57.721926 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:57.721894    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:57.842909 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:57.842865    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:58.016622 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:58.016577    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:59.411694 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.411662    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"]
Oct 30 00:52:59.415787 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.415759    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.434525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.434497    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"]
Oct 30 00:52:59.445516 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.445491    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"premium-w5-da76a2-kserve-self-signed-certs\""
Oct 30 00:52:59.446872 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.446857    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-dshm\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.446942 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.446903    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cfa548-1c34-4604-9013-acbf1b99b3ed-tls-certs\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.447038 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.447017    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-home\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.447099 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.447051    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-tmp-dir\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.447099 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.447084    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdfc6\" (UniqueName: \"kubernetes.io/projected/31cfa548-1c34-4604-9013-acbf1b99b3ed-kube-api-access-cdfc6\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.447204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.447138    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-model-cache\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.488450 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.488416    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:59.547777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.547744    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-dshm\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.547971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.547799    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cfa548-1c34-4604-9013-acbf1b99b3ed-tls-certs\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.547971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.547830    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-home\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.547971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.547849    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-tmp-dir\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.547971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.547868    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfc6\" (UniqueName: \"kubernetes.io/projected/31cfa548-1c34-4604-9013-acbf1b99b3ed-kube-api-access-cdfc6\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.547971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.547889    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-model-cache\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.548331 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.548306    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-home\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.548511 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.548337    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-model-cache\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.548628 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.548515    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-tmp-dir\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.550538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.550515    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-dshm\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.550765 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.550749    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cfa548-1c34-4604-9013-acbf1b99b3ed-tls-certs\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.569025 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.568998    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfc6\" (UniqueName: \"kubernetes.io/projected/31cfa548-1c34-4604-9013-acbf1b99b3ed-kube-api-access-cdfc6\") pod \"premium-w5-da76a2-kserve-668bb45f74-zfp2n\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") " pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:52:59.670647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.670550    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:52:59.731295 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:52:59.731257    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:53:00.032914 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.032892    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"]
Oct 30 00:53:00.035403 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:53:00.035374    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31cfa548_1c34_4604_9013_acbf1b99b3ed.slice/crio-21d20126c0cff1231f8926584b008d0072722247827a2139e80ae71e465a165d WatchSource:0}: Error finding container 21d20126c0cff1231f8926584b008d0072722247827a2139e80ae71e465a165d: Status 404 returned error can't find the container with id 21d20126c0cff1231f8926584b008d0072722247827a2139e80ae71e465a165d
Oct 30 00:53:00.081125 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.081092    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"]
Oct 30 00:53:00.084654 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.084631    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.110332 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.110306    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"]
Oct 30 00:53:00.115642 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.115611    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8cf33a-openshift-default-dockercfg-j7d2l\""
Oct 30 00:53:00.153302 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153269    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.153488 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153343    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.153488 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153429    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.153488 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153467    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44mp\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-kube-api-access-p44mp\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.153622 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153537    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.153622 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153562    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.153704 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153623    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.153704 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153639    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.153704 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.153658    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100b43f2-ffa5-4095-ab1a-bbcb77417026-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255182 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255145    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255407 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255220    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255407 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255286    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255407 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255334    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44mp\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-kube-api-access-p44mp\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255407 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255397    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255627 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255429    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255627 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255520    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255627 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255546    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255627 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255621    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255835 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255761    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255835 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255787    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255939 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255847    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.255939 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.255695    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100b43f2-ffa5-4095-ab1a-bbcb77417026-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.256247 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.256211    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100b43f2-ffa5-4095-ab1a-bbcb77417026-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.257758 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.257734    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.257865 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.257849    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.276696 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.276669    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.276818 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.276805    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44mp\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-kube-api-access-p44mp\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.397932 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.397849    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:00.648857 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.648777    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" event={"ID":"31cfa548-1c34-4604-9013-acbf1b99b3ed","Type":"ContainerStarted","Data":"7adefbd1cbf535cb4a596ec88abf2adc5fc4fc882456facb14b3b8d8b5bf8f47"}
Oct 30 00:53:00.648857 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.648821    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" event={"ID":"31cfa548-1c34-4604-9013-acbf1b99b3ed","Type":"ContainerStarted","Data":"21d20126c0cff1231f8926584b008d0072722247827a2139e80ae71e465a165d"}
Oct 30 00:53:00.691837 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.691789    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podStartSLOduration=1.691774604 podStartE2EDuration="1.691774604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:00.691106095 +0000 UTC m=+860.762429928" watchObservedRunningTime="2026-10-30 00:53:00.691774604 +0000 UTC m=+860.763098456"
Oct 30 00:53:00.718063 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.718040    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"]
Oct 30 00:53:00.720288 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:53:00.720259    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100b43f2_ffa5_4095_ab1a_bbcb77417026.slice/crio-b10516fb7bd0fd471ac7de0439ca1d2624205cf06709e037f462b043994b3f40 WatchSource:0}: Error finding container b10516fb7bd0fd471ac7de0439ca1d2624205cf06709e037f462b043994b3f40: Status 404 returned error can't find the container with id b10516fb7bd0fd471ac7de0439ca1d2624205cf06709e037f462b043994b3f40
Oct 30 00:53:00.722849 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.722818    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:00.722950 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.722882    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:00.722950 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.722914    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:00.726603 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.726582    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"]
Oct 30 00:53:00.732093 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.732076    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:53:00.732197 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.732147    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.755568 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.755267    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"]
Oct 30 00:53:00.756925 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.756897    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"]
Oct 30 00:53:00.760510 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760470    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.760642 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760514    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.760642 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760601    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbhzc\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-kube-api-access-lbhzc\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.760746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760654    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.760746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760687    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.760746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760738    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.760887 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760778    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.760887 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760841    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.760887 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.760875    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.761310 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.761293    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.783076 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.783047    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"]
Oct 30 00:53:00.862175 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862126    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.862364 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862271    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.862364 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862303    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.862364 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862332    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.862364 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862360    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.862534 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862393    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.862534 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862458    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzc\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-kube-api-access-lbhzc\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.862534 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862513    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.862671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862537    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrkr\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-kube-api-access-tjrkr\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.862671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862573    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.862671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862626    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.862671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862665    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.862800 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862689    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.862800 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862715    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.862800 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862726    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.862800 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.862746    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.863105 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.863070    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.863525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.863446    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.863525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.863443    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.863525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.863487    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.863676 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.863558    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.863676 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.863602    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.863676 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.863560    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.865711 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.865682    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.866031 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.866005    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.883705 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.883677    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbhzc\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-kube-api-access-lbhzc\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.884521 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.884487    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:00.964916 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.964878    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.964916 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.964919    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.964939    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965007    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrkr\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-kube-api-access-tjrkr\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965073    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965097    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965126    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965157    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965520 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965216    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965575 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965555    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965644    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-credential-socket\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965757 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965739    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-data\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965919 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965882    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-certs\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.965987 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.965961    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istiod-ca-cert\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.967407 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.967387    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-envoy\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.967887 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.967865    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-podinfo\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.988592 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.988559    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-token\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:00.988734 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:00.988667    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrkr\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-kube-api-access-tjrkr\") pod \"e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") " pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:01.063505 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.063475    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:01.095812 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.095754    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:01.360261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.360162    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:01.389148 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.389126    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"]
Oct 30 00:53:01.391285 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:53:01.391222    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea14f71_1fc9_426a_9ed4_8148160ff3aa.slice/crio-6a13af3bce5a734ad3fb12ccf8de71ba729381d1d2ddc008dce5a6ba4b7890ec WatchSource:0}: Error finding container 6a13af3bce5a734ad3fb12ccf8de71ba729381d1d2ddc008dce5a6ba4b7890ec: Status 404 returned error can't find the container with id 6a13af3bce5a734ad3fb12ccf8de71ba729381d1d2ddc008dce5a6ba4b7890ec
Oct 30 00:53:01.394030 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.393996    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:01.394123 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.394064    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:01.394123 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.394101    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:01.440887 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.440864    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"]
Oct 30 00:53:01.446359 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:53:01.446334    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae9e956_89af_429f_9b02_a3dcc36d9c14.slice/crio-651fcedb7dda4fae1c7b80c8fa37c9d3ea8dc6f2946b80145e459da7b09267a3 WatchSource:0}: Error finding container 651fcedb7dda4fae1c7b80c8fa37c9d3ea8dc6f2946b80145e459da7b09267a3: Status 404 returned error can't find the container with id 651fcedb7dda4fae1c7b80c8fa37c9d3ea8dc6f2946b80145e459da7b09267a3
Oct 30 00:53:01.448868 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.448839    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:01.448931 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.448907    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:01.448969 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.448935    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:01.658444 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.658352    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" event={"ID":"9ae9e956-89af-429f-9b02-a3dcc36d9c14","Type":"ContainerStarted","Data":"4f00ca18e461c310b6834c19b0798b6d48b9ad7772a6a5cf0eb61e7c25fb1a62"}
Oct 30 00:53:01.659041 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.659000    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" event={"ID":"9ae9e956-89af-429f-9b02-a3dcc36d9c14","Type":"ContainerStarted","Data":"651fcedb7dda4fae1c7b80c8fa37c9d3ea8dc6f2946b80145e459da7b09267a3"}
Oct 30 00:53:01.662320 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.662292    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" event={"ID":"cea14f71-1fc9-426a-9ed4-8148160ff3aa","Type":"ContainerStarted","Data":"18ed205a8bffeeba972efab4c7763351d6f56ff28264ec4943e1a1cc424eda36"}
Oct 30 00:53:01.662507 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.662488    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" event={"ID":"cea14f71-1fc9-426a-9ed4-8148160ff3aa","Type":"ContainerStarted","Data":"6a13af3bce5a734ad3fb12ccf8de71ba729381d1d2ddc008dce5a6ba4b7890ec"}
Oct 30 00:53:01.668916 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.668768    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" event={"ID":"100b43f2-ffa5-4095-ab1a-bbcb77417026","Type":"ContainerStarted","Data":"879d87154e408a4e288b31d9ff675a57001d4b9b31a097884a6a695418ebd1dd"}
Oct 30 00:53:01.668916 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.668813    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" event={"ID":"100b43f2-ffa5-4095-ab1a-bbcb77417026","Type":"ContainerStarted","Data":"b10516fb7bd0fd471ac7de0439ca1d2624205cf06709e037f462b043994b3f40"}
Oct 30 00:53:01.763034 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.762930    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" podStartSLOduration=1.7629107739999998 podStartE2EDuration="1.762910774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:01.715761964 +0000 UTC m=+861.787085798" watchObservedRunningTime="2026-10-30 00:53:01.762910774 +0000 UTC m=+861.834234610"
Oct 30 00:53:01.764302 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.764262    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" podStartSLOduration=1.764250891 podStartE2EDuration="1.764250891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:01.763025332 +0000 UTC m=+861.834349144" watchObservedRunningTime="2026-10-30 00:53:01.764250891 +0000 UTC m=+861.835574796"
Oct 30 00:53:01.816128 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:01.816069    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podStartSLOduration=1.816051895 podStartE2EDuration="1.816051895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:01.813726172 +0000 UTC m=+861.885050008" watchObservedRunningTime="2026-10-30 00:53:01.816051895 +0000 UTC m=+861.887375729"
Oct 30 00:53:02.063696 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.063661    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:02.064974 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.064954    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:53:02.065059 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.065041    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:53:02.120267 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.116940    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:53:02.120267 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.116996    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 00:53:02.144290 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.132390    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:02.402316 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.401223    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:02.406258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.403266    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:53:02.406258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:02.403318    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 00:53:03.064618 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:03.064584    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:53:03.067091 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:03.065122    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:53:03.101783 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:03.101750    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:53:03.102089 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:03.102039    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 00:53:03.133327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:03.133128    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:03.404323 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:03.400258    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:53:03.404323 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:03.400316    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 00:53:04.064639 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:04.064595    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:53:04.065028 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:04.064653    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:53:04.100257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:04.099693    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:53:04.100257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:04.099769    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 00:53:04.405326 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:04.404663    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:53:04.405326 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:04.404733    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 00:53:04.734258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:04.733731    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:05.065044 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:05.064968    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:53:05.065044 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:05.065017    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:53:05.096428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:05.096392    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:53:05.096699 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:05.096672    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 00:53:05.398499 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:05.398401    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:53:05.398757 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:05.398730    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 00:53:06.064314 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:06.064277    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:53:06.064509 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:06.064341    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:53:06.099629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:06.099594    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:53:06.121181 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:06.121146    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 00:53:06.397314 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:06.396867    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:06.401258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:06.398345    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:53:06.401258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:06.398398    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 00:53:07.064750 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:07.064721    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 00:53:07.064988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:07.064961    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 00:53:07.099254 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:07.098796    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 00:53:07.099254 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:07.098875    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 00:53:07.398518 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:07.398413    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 00:53:07.399075 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:07.399044    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 00:53:07.976258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:07.976197    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:08.149847 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.149825    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:08.152701 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.152674    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:08.154247 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.154215    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 00:53:08.218121 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.218091    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:08.218615 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.218593    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:08.220076 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.220054    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 00:53:08.305447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.305343    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:53:08.402765 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.402719    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:08.403187 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.402969    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:08.403774 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:08.403756    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 00:53:09.693558 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:09.693515    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:09.731512 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:09.731484    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:53:09.744857 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:09.744833    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:53:09.781944 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:09.781917    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:53:10.163721 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:10.163625    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:11.473375 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:11.473334    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:12.204425 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:12.204389    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:12.954908 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:12.954877    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:14.576518 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:14.576480    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:16.251699 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:16.251664    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:17.853195 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:17.853150    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:18.395017 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:53:18.394971    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:53:19.949513 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:19.949463    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:20.936050 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:20.936010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:21.692894 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:21.692863    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:23.254630 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:23.254592    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:24.706692 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:24.706652    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:26.249933 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:26.249895    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:27.186627 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:27.186590    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:27.668217 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:27.668185    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:29.085961 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:29.085925    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:53:29.275301 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:29.275267    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:30.224531 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.224492    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:30.641868 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.641771    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"]
Oct 30 00:53:30.646037 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.646014    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.664573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.664547    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-da76a2-openshift-default-dockercfg-8l8nk\""
Oct 30 00:53:30.680365 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.680330    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"]
Oct 30 00:53:30.727294 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727253    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.727294 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727285    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.727554 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727377    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.727554 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727406    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.727554 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727441    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.727554 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727526    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.727759 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727566    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.727759 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727586    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.727759 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.727616    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmz2x\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-kube-api-access-fmz2x\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.828541 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.828509    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.828751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.828548    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.828751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.828577    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmz2x\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-kube-api-access-fmz2x\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.828751 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.828681    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.828917 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.828795    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.828917 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.828891    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.829019 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.828916    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.829019 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.829000    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.829279 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.829251    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.829436 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.829402    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.829512 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.829478    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.829789 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.829765    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.830001 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.829979    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.830065 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.830032    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.831435 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.831403    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.832349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.832324    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.850075 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.850049    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.850345 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.850324    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmz2x\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-kube-api-access-fmz2x\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:30.962683 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:30.962643    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:31.293709 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.293681    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"]
Oct 30 00:53:31.296005 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:53:31.295970    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfe27d5_73b3_4988_91dc_87d3de6b88d5.slice/crio-6cdaf572e08be852c5eb9e8ffd02d56562260788643a59639589018c565aa18c WatchSource:0}: Error finding container 6cdaf572e08be852c5eb9e8ffd02d56562260788643a59639589018c565aa18c: Status 404 returned error can't find the container with id 6cdaf572e08be852c5eb9e8ffd02d56562260788643a59639589018c565aa18c
Oct 30 00:53:31.298580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.298551    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:31.298668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.298628    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:31.298668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.298664    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:31.688213 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.688182    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:31.906606 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.906564    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" event={"ID":"bdfe27d5-73b3-4988-91dc-87d3de6b88d5","Type":"ContainerStarted","Data":"a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57"}
Oct 30 00:53:31.906786 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.906632    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" event={"ID":"bdfe27d5-73b3-4988-91dc-87d3de6b88d5","Type":"ContainerStarted","Data":"6cdaf572e08be852c5eb9e8ffd02d56562260788643a59639589018c565aa18c"}
Oct 30 00:53:31.963163 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.963071    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:31.979549 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:31.979493    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" podStartSLOduration=1.979476915 podStartE2EDuration="1.979476915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:31.964794753 +0000 UTC m=+892.036118589" watchObservedRunningTime="2026-10-30 00:53:31.979476915 +0000 UTC m=+892.050800833"
Oct 30 00:53:32.965849 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:32.965808    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:32.966286 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:32.965879    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 00:53:33.193556 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:33.193398    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:33.963594 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:33.963554    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:33.963780 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:33.963620    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 00:53:34.794902 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:34.794861    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:34.963921 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:34.963893    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:34.964101 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:34.963944    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 00:53:35.197548 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.197514    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"]
Oct 30 00:53:35.201268 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.201249    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.220011 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.219981    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"]
Oct 30 00:53:35.231062 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.231034    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:53:35.231260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.231162    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 00:53:35.309692 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.309654    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.309865 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.309818    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddsh\" (UniqueName: \"kubernetes.io/projected/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-kube-api-access-zddsh\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.309911 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.309892    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.411475 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.411435    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddsh\" (UniqueName: \"kubernetes.io/projected/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-kube-api-access-zddsh\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.411630 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.411514    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.411630 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.411554    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.412296 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.412261    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.414095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.414066    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.432829 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.432798    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddsh\" (UniqueName: \"kubernetes.io/projected/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-kube-api-access-zddsh\") pod \"maas-api-revoke-sub-00335efb-g2phd\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.515458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.515383    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:35.835685 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.835661    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"]
Oct 30 00:53:35.837990 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:53:35.837959    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd092b5_f93a_4f2f_9482_c0c4e242fa7b.slice/crio-a04314298e63e010a51e2a0092d1836346dedac65d5a6b5204adfa47ca0cec23 WatchSource:0}: Error finding container a04314298e63e010a51e2a0092d1836346dedac65d5a6b5204adfa47ca0cec23: Status 404 returned error can't find the container with id a04314298e63e010a51e2a0092d1836346dedac65d5a6b5204adfa47ca0cec23
Oct 30 00:53:35.928456 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.928415    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd" event={"ID":"efd092b5-f93a-4f2f-9482-c0c4e242fa7b","Type":"ContainerStarted","Data":"a04314298e63e010a51e2a0092d1836346dedac65d5a6b5204adfa47ca0cec23"}
Oct 30 00:53:35.964103 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.964071    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:35.964294 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:35.964123    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 00:53:36.465036 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:36.464991    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:36.963014 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:36.962965    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:36.963517 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:36.963037    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 00:53:37.935749 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:37.935713    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:37.963945 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:37.963894    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 00:53:37.964612 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:37.963969    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 00:53:38.952081 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:38.952020    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd" event={"ID":"efd092b5-f93a-4f2f-9482-c0c4e242fa7b","Type":"ContainerStarted","Data":"851b7a555c05197d83b56d258a583915a28da706e5e245a5085f4b7c63fdc774"}
Oct 30 00:53:38.968047 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:38.968016    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:38.968599 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:38.968469    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:38.969266 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:38.969222    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 00:53:38.995543 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:38.995496    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd" podStartSLOduration=1.50915403 podStartE2EDuration="3.995483097s" totalImagesPullingTime="2.486329067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:38.993885333 +0000 UTC m=+899.065209192" watchObservedRunningTime="2026-10-30 00:53:38.995483097 +0000 UTC m=+899.066806932"
Oct 30 00:53:39.358073 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:39.357980    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:39.958905 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:39.958866    4576 generic.go:356] "Generic (PLEG): container finished" podID="efd092b5-f93a-4f2f-9482-c0c4e242fa7b" containerID="851b7a555c05197d83b56d258a583915a28da706e5e245a5085f4b7c63fdc774" exitCode=0
Oct 30 00:53:39.959076 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:39.958956    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd" event={"ID":"efd092b5-f93a-4f2f-9482-c0c4e242fa7b","Type":"ContainerDied","Data":"851b7a555c05197d83b56d258a583915a28da706e5e245a5085f4b7c63fdc774"}
Oct 30 00:53:40.892258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:40.892195    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:41.120699 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.120670    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:41.183041 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.182946    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-xks-ca\") pod \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") "
Oct 30 00:53:41.183219 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.183055    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-kube-api-access-zddsh\" (UniqueName: \"kubernetes.io/projected/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-kube-api-access-zddsh\") pod \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") "
Oct 30 00:53:41.183219 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.183150    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-openshift-ca\") pod \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\" (UID: \"efd092b5-f93a-4f2f-9482-c0c4e242fa7b\") "
Oct 30 00:53:41.183695 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.183673    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-openshift-ca" pod "efd092b5-f93a-4f2f-9482-c0c4e242fa7b" (UID: "efd092b5-f93a-4f2f-9482-c0c4e242fa7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:41.185455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.185420    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-xks-ca" pod "efd092b5-f93a-4f2f-9482-c0c4e242fa7b" (UID: "efd092b5-f93a-4f2f-9482-c0c4e242fa7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:41.185562 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.185521    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-kube-api-access-zddsh" pod "efd092b5-f93a-4f2f-9482-c0c4e242fa7b" (UID: "efd092b5-f93a-4f2f-9482-c0c4e242fa7b"). InnerVolumeSpecName "kube-api-access-zddsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:41.284412 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.284378    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:53:41.284412 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.284407    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:53:41.284631 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.284422    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddsh\" (UniqueName: \"kubernetes.io/projected/efd092b5-f93a-4f2f-9482-c0c4e242fa7b-kube-api-access-zddsh\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:53:41.971194 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.971153    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd" event={"ID":"efd092b5-f93a-4f2f-9482-c0c4e242fa7b","Type":"ContainerDied","Data":"a04314298e63e010a51e2a0092d1836346dedac65d5a6b5204adfa47ca0cec23"}
Oct 30 00:53:41.971632 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.971166    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"
Oct 30 00:53:41.971632 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:41.971226    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a04314298e63e010a51e2a0092d1836346dedac65d5a6b5204adfa47ca0cec23"
Oct 30 00:53:42.735992 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:42.735955    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:44.953778 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:44.953738    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:45.350010 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.349928    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"]
Oct 30 00:53:45.350400 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.350386    4576 cpu_manager.go:414] "containerMap: removing container" podUID="efd092b5-f93a-4f2f-9482-c0c4e242fa7b" containerName="revoke-keys"
Oct 30 00:53:45.350468 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.350401    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd092b5-f93a-4f2f-9482-c0c4e242fa7b" containerName="revoke-keys"
Oct 30 00:53:45.350508 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.350497    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd092b5-f93a-4f2f-9482-c0c4e242fa7b" containerName="revoke-keys"
Oct 30 00:53:45.353835 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.353820    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.386964 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.386931    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3c2596-openshift-default-dockercfg-mts5l\""
Oct 30 00:53:45.389029 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.389007    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"]
Oct 30 00:53:45.460565 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460531    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xqr\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-kube-api-access-92xqr\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.460743 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460577    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0987db5-792d-4300-a535-2dd9b7a17cb3-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.460743 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460602    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.460844 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460770    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.460844 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460801    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.460844 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460819    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.460844 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460838    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.460990 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460982    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.461021 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.460998    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.561847 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.561809    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.562113 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562088    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.562269 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562174    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xqr\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-kube-api-access-92xqr\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.562269 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562213    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0987db5-792d-4300-a535-2dd9b7a17cb3-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.562400 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562272    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.562400 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562377    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.562481 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562410    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.562481 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562441    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.562481 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562471    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.563090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562863    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.563090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.562960    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.563090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.563153    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.563818 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.563345    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0987db5-792d-4300-a535-2dd9b7a17cb3-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.563818 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.563405    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.565273 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.565227    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.565969 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.565950    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.566544 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.566526    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"]
Oct 30 00:53:45.571123 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.571109    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.598576 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.598548    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.603550 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.603483    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xqr\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-kube-api-access-92xqr\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.607159 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.607128    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b565ec-openshift-default-dockercfg-sl86j\""
Oct 30 00:53:45.608156 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.608131    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"]
Oct 30 00:53:45.663905 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.663874    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.663905 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.663908    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.664158 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.664078    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.664222 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.664167    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv9df\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-kube-api-access-dv9df\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.664307 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.664290    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.664343 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.664323    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.664398 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.664346    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.664471 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.664452    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.664531 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.664504    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.666707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.666684    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:45.766084 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766053    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766084 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766094    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766370 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766115    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766517 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766487    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766633 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766558    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766633 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766557    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766633 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766595    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766633 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766625    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766852 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766725    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766852 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766736    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.766852 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766834    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.767066 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.766970    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.767130 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.767080    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.767130 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.767092    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv9df\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-kube-api-access-dv9df\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.769226 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.769203    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.769364 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.769329    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.786939 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.786920    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.787437 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.787421    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv9df\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-kube-api-access-dv9df\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.893971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.893899    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:45.976854 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.976822    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"]
Oct 30 00:53:45.978669 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:53:45.978606    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0987db5_792d_4300_a535_2dd9b7a17cb3.slice/crio-b5d07ce6b80863e41819f57e418f2908752bf137a57801facc616bff49a6f2fa WatchSource:0}: Error finding container b5d07ce6b80863e41819f57e418f2908752bf137a57801facc616bff49a6f2fa: Status 404 returned error can't find the container with id b5d07ce6b80863e41819f57e418f2908752bf137a57801facc616bff49a6f2fa
Oct 30 00:53:45.981529 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.981500    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:45.981630 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.981567    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:45.981630 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:45.981598    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:46.000035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:46.000005    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" event={"ID":"f0987db5-792d-4300-a535-2dd9b7a17cb3","Type":"ContainerStarted","Data":"b5d07ce6b80863e41819f57e418f2908752bf137a57801facc616bff49a6f2fa"}
Oct 30 00:53:46.195734 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:46.195707    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"]
Oct 30 00:53:46.197993 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:53:46.197964    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3049ef_0be4_4a1b_8c6d_d05d5e81b1b4.slice/crio-3d90b7da09fcb1bc5858127e11c8484eebfb4f1ee4a5b7364a3e338c137a40ff WatchSource:0}: Error finding container 3d90b7da09fcb1bc5858127e11c8484eebfb4f1ee4a5b7364a3e338c137a40ff: Status 404 returned error can't find the container with id 3d90b7da09fcb1bc5858127e11c8484eebfb4f1ee4a5b7364a3e338c137a40ff
Oct 30 00:53:46.200434 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:46.200405    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:46.200519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:46.200469    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:46.200519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:46.200498    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:53:46.684311 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:46.683834    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:47.023991 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.023949    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" event={"ID":"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4","Type":"ContainerStarted","Data":"fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4"}
Oct 30 00:53:47.024557 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.024530    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" event={"ID":"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4","Type":"ContainerStarted","Data":"3d90b7da09fcb1bc5858127e11c8484eebfb4f1ee4a5b7364a3e338c137a40ff"}
Oct 30 00:53:47.026743 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.026718    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" event={"ID":"f0987db5-792d-4300-a535-2dd9b7a17cb3","Type":"ContainerStarted","Data":"7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f"}
Oct 30 00:53:47.107259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.107083    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" podStartSLOduration=2.107065534 podStartE2EDuration="2.107065534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:47.092294707 +0000 UTC m=+907.163618542" watchObservedRunningTime="2026-10-30 00:53:47.107065534 +0000 UTC m=+907.178389370"
Oct 30 00:53:47.151560 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.151501    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" podStartSLOduration=2.151480652 podStartE2EDuration="2.151480652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:47.148601901 +0000 UTC m=+907.219925754" watchObservedRunningTime="2026-10-30 00:53:47.151480652 +0000 UTC m=+907.222804489"
Oct 30 00:53:47.667201 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.667173    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:47.675258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.672872    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:53:47.675258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.672928    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 00:53:47.897257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.896353    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:47.897257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.896516    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:53:47.897257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:47.896561    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 00:53:48.220086 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:48.220044    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:48.429080 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:53:48.429046    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:53:48.691308 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:48.689682    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:53:48.691308 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:48.689745    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 00:53:48.901256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:48.900700    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:53:48.901256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:48.900767    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 00:53:49.667013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:49.666984    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:53:49.667486 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:49.667036    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 00:53:49.859501 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:49.859466    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:49.897886 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:49.897840    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:53:49.898206 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:49.898166    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 00:53:50.697260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:50.676472    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:53:50.697864 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:50.697832    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 00:53:50.897269 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:50.896708    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:53:50.897269 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:50.896773    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 00:53:51.301458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:51.301422    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:51.667162 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:51.667087    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 00:53:51.667162 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:51.667137    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 00:53:51.906258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:51.905613    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 00:53:51.906258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:51.905674    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 00:53:51.996754 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:51.996715    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:53:52.820651 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:52.820610    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:52.850825 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:52.850793    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:52.852345 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:52.852302    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:52.853547 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:52.853513    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 00:53:53.922724 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:53.922680    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:53:53.923123 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:53.922745    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:53.924030 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:53.924011    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:53.924291 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:53.924265    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:53.925216 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:53.925196    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 00:53:54.470927 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:54.470873    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:55.936538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:55.936509    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:57.562798 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:57.562758    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:53:59.383616 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:53:59.383576    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:00.108746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.108714    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"]
Oct 30 00:54:00.114729 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.114706    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.133318 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.133279    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7d2b75-openshift-default-dockercfg-pft72\""
Oct 30 00:54:00.149335 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.149291    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"]
Oct 30 00:54:00.219902 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.219864    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-envoy\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.220090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.219923    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8207408b-42d0-41ef-887c-cddea6ccfb64-istiod-ca-cert\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.220090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.220072    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-token\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.220161 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.220094    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-podinfo\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.220365 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.220333    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-data\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.220488 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.220383    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-credential-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.220488 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.220464    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp8sq\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-kube-api-access-wp8sq\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.220628 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.220567    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-certs\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.220628 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.220620    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.321768 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.321732    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-data\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.321768 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.321775    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-credential-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322067 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.321903    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp8sq\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-kube-api-access-wp8sq\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322067 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.321987    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-certs\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322067 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322016    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322067 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322052    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-envoy\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322110    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8207408b-42d0-41ef-887c-cddea6ccfb64-istiod-ca-cert\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322121    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-credential-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322187    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-token\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322211    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-data\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322221    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-podinfo\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322386    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-socket\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322413    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-certs\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.322706 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.322685    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8207408b-42d0-41ef-887c-cddea6ccfb64-istiod-ca-cert\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.324474 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.324449    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-envoy\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.324741 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.324723    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-podinfo\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.342850 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.342820    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp8sq\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-kube-api-access-wp8sq\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.343160 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.343140    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-token\") pod \"e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") " pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.428885 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.428784    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:00.753877 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.753851    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"]
Oct 30 00:54:00.755341 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:54:00.755312    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8207408b_42d0_41ef_887c_cddea6ccfb64.slice/crio-942323e7c18f030159dc03db617d7646461636af01616f8547f6665c7cd3b9c3 WatchSource:0}: Error finding container 942323e7c18f030159dc03db617d7646461636af01616f8547f6665c7cd3b9c3: Status 404 returned error can't find the container with id 942323e7c18f030159dc03db617d7646461636af01616f8547f6665c7cd3b9c3
Oct 30 00:54:00.757847 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.757818    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:00.757906 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.757883    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:00.757946 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.757913    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:00.960805 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:00.960757    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:01.146314 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:01.146200    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" event={"ID":"8207408b-42d0-41ef-887c-cddea6ccfb64","Type":"ContainerStarted","Data":"a9469f134609370b0f0893d5feb5c7383e73056abf4006fbe8d3d91a6b68d069"}
Oct 30 00:54:01.146314 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:01.146279    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" event={"ID":"8207408b-42d0-41ef-887c-cddea6ccfb64","Type":"ContainerStarted","Data":"942323e7c18f030159dc03db617d7646461636af01616f8547f6665c7cd3b9c3"}
Oct 30 00:54:01.190056 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:01.190006    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" podStartSLOduration=1.189993958 podStartE2EDuration="1.189993958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:01.188703946 +0000 UTC m=+921.260027780" watchObservedRunningTime="2026-10-30 00:54:01.189993958 +0000 UTC m=+921.261317792"
Oct 30 00:54:01.429362 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:01.429273    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:01.431207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:01.431183    4576 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 00:54:01.431342 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:01.431245    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 00:54:02.429635 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:02.429600    4576 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 00:54:02.430036 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:02.429666    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 00:54:02.698530 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:02.698499    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:03.429629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:03.429591    4576 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 00:54:03.429785 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:03.429645    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 00:54:04.393952 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:04.393910    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:04.429511 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:04.429478    4576 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 00:54:04.429695 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:04.429555    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 00:54:05.429992 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:05.429953    4576 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 00:54:05.430424 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:05.430023    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 00:54:06.235407 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:06.235373    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:06.429656 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:06.429614    4576 patch_prober.go:29] interesting pod/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 00:54:06.429837 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:06.429681    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 00:54:07.582803 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:07.582765    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:07.583428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:07.583397    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:07.584548 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:07.584529    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 00:54:07.773551 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:07.773515    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:09.689150 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:09.689113    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:11.590805 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:11.590771    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:13.349830 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:13.349790    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:13.834088 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:13.834052    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:15.685207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:15.685174    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:17.276428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:17.276375    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:18.381223 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:54:18.381190    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:54:18.722545 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:18.722507    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:20.377400 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:20.377354    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:21.816162 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:21.816077    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:22.399993 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:22.399964    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:54:23.497633 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:23.497577    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:26.394260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:26.393871    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:28.060726 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:28.060687    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:28.335180 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:28.335101    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:28.680256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:28.680152    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:29.661569 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:29.661525    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:30.337131 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.337096    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"]
Oct 30 00:54:30.340972 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.340951    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.361620 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.361592    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"]
Oct 30 00:54:30.367149 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367121    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.367349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367162    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.367349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367203    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.367349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367224    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.367349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367310    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.367349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367333    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.367630 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367440    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.367630 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367467    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftssz\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-kube-api-access-ftssz\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.367630 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.367512    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.443870 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.443837    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:30.468486 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.468452    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.468486 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.468494    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftssz\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-kube-api-access-ftssz\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.468737 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.468555    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.468737 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.468705    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.468737 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.468731    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.468900 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.468769    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.468900 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.468820    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-data\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.469141 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.469118    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-credential-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.469345 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.468791    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.469498 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.469471    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-certs\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.469586 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.469370    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-socket\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.469586 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.469511    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.469586 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.469543    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.470416 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.470397    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istiod-ca-cert\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.471863 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.471840    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-podinfo\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.472272 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.472219    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-envoy\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.496573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.496544    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftssz\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-kube-api-access-ftssz\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.497427 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.497408    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-token\") pod \"e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") " pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.658100 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.658029    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:30.983103 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.983022    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"]
Oct 30 00:54:30.987578 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:54:30.987539    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d71d034_e6e6_4133_be6f_a22b75bd12d9.slice/crio-3583114197dce765666117f2510fd79a211e37be89ca340686c8738abb1ad5ab WatchSource:0}: Error finding container 3583114197dce765666117f2510fd79a211e37be89ca340686c8738abb1ad5ab: Status 404 returned error can't find the container with id 3583114197dce765666117f2510fd79a211e37be89ca340686c8738abb1ad5ab
Oct 30 00:54:30.990378 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.990341    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:30.990491 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.990414    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:30.990491 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:30.990452    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:31.196694 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.196658    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:31.481536 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.481488    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" event={"ID":"6d71d034-e6e6-4133-be6f-a22b75bd12d9","Type":"ContainerStarted","Data":"30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578"}
Oct 30 00:54:31.481798 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.481547    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" event={"ID":"6d71d034-e6e6-4133-be6f-a22b75bd12d9","Type":"ContainerStarted","Data":"3583114197dce765666117f2510fd79a211e37be89ca340686c8738abb1ad5ab"}
Oct 30 00:54:31.540549 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.540499    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" podStartSLOduration=1.540484886 podStartE2EDuration="1.540484886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:31.538585537 +0000 UTC m=+951.609909370" watchObservedRunningTime="2026-10-30 00:54:31.540484886 +0000 UTC m=+951.611808721"
Oct 30 00:54:31.659062 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.659027    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:31.842692 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.842611    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"]
Oct 30 00:54:31.846765 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.846739    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.870546 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.870515    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"]
Oct 30 00:54:31.907291 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.906962    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4n6\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-kube-api-access-8b4n6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.907291 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.907045    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.907291 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.907097    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.907291 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.907135    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.907291 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.907177    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.908858 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.907675    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.908858 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.907727    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91dc086b-92dc-48ac-a0e1-59354998ab33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.908858 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.907765    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:31.908858 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:31.907820    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009043 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009006    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009043 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009060    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009086    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009116    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009150    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009169    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91dc086b-92dc-48ac-a0e1-59354998ab33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009197    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009279    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009348    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4n6\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-kube-api-access-8b4n6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.009647 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009589    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.010027 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009796    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.010027 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.009966    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.010283 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.010262    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.010362 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.010296    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91dc086b-92dc-48ac-a0e1-59354998ab33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.011926 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.011904    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.012048 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.011986    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.029984 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.029963    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.030123 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.030106    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4n6\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-kube-api-access-8b4n6\") pod \"maas-default-gateway-openshift-default-6db586dd6c-fjp5b\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.161723 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.161649    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:32.263110 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.263002    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:32.404197 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.404118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:32.535873 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.534717    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"]
Oct 30 00:54:32.541913 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.541694    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:32.541913 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.541777    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:32.541913 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.541821    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:54:32.660467 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.660435    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:54:32.660576 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:32.660479    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 00:54:33.321055 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:33.320467    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:33.528305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:33.528265    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" event={"ID":"91dc086b-92dc-48ac-a0e1-59354998ab33","Type":"ContainerStarted","Data":"d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992"}
Oct 30 00:54:33.528545 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:33.528520    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" event={"ID":"91dc086b-92dc-48ac-a0e1-59354998ab33","Type":"ContainerStarted","Data":"81665df1c1e12b759fdd0c1937a911df8dfc999ae0028c8331b6d81634c4cc19"}
Oct 30 00:54:33.608318 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:33.604565    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podStartSLOduration=2.604531916 podStartE2EDuration="2.604531916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:33.585743889 +0000 UTC m=+953.657067736" watchObservedRunningTime="2026-10-30 00:54:33.604531916 +0000 UTC m=+953.675855752"
Oct 30 00:54:33.666258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:33.659900    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:54:33.666258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:33.659970    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 00:54:34.174260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:34.170793    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:34.174260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:34.170958    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:54:34.174260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:34.171003    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:54:34.659691 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:34.659658    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:54:34.660309 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:34.660276    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 00:54:35.162162 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:35.162124    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:54:35.162379 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:35.162178    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:54:35.658463 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:35.658415    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:54:35.658692 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:35.658473    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 00:54:35.748748 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:35.748712    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:36.162417 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:36.162384    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:54:36.162594 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:36.162439    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:54:36.658530 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:36.658492    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:54:36.658707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:36.658550    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 00:54:37.162357 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:37.162321    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:54:37.162749 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:37.162383    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:54:37.619686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:37.619646    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:37.659329 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:37.659296    4576 patch_prober.go:29] interesting pod/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 00:54:37.659497 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:37.659360    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 00:54:38.162523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:38.162487    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:54:38.162919 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:38.162544    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:54:38.662719 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:38.662690    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:38.663049 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:38.663030    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:38.663770 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:38.663748    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 00:54:39.162598 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:39.162563    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:54:39.163010 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:39.162624    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:54:39.611589 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:39.611554    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:40.162290 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:40.162251    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:54:40.162466 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:40.162310    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:54:41.242728 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:41.242699    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:41.243214 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:41.243073    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:41.244053 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:41.244034    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 00:54:41.359179 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:41.359143    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:42.403452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:42.403416    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:54:42.873517 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:42.873475    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:44.472906 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:44.472870    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:45.936580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:45.936550    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:46.003988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.003957    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"]
Oct 30 00:54:46.009834 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.009805    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.025720 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.025657    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"]
Oct 30 00:54:46.036609 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.036581    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-da76a2\"/\"distinct-w5-da76a2-kserve-self-signed-certs\""
Oct 30 00:54:46.083910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.083868    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:46.154130 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.154094    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9gxq\" (UniqueName: \"kubernetes.io/projected/6ad4cd2d-6793-4856-826b-bcb46811db0d-kube-api-access-d9gxq\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.154352 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.154138    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-dshm\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.154352 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.154160    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-tmp-dir\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.154479 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.154403    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-model-cache\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.154479 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.154460    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-home\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.154623 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.154603    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4cd2d-6793-4856-826b-bcb46811db0d-tls-certs\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.256035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.255994    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9gxq\" (UniqueName: \"kubernetes.io/projected/6ad4cd2d-6793-4856-826b-bcb46811db0d-kube-api-access-d9gxq\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.256267 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.256066    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-dshm\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.256267 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.256097    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-tmp-dir\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.256267 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.256180    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-model-cache\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.256267 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.256226    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-home\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.256501 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.256294    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4cd2d-6793-4856-826b-bcb46811db0d-tls-certs\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.257258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.257196    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-model-cache\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.257550 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.257516    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-tmp-dir\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.257674 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.257516    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-home\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.260893 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.260865    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-dshm\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.264015 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.263994    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4cd2d-6793-4856-826b-bcb46811db0d-tls-certs\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.283495 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.282747    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9gxq\" (UniqueName: \"kubernetes.io/projected/6ad4cd2d-6793-4856-826b-bcb46811db0d-kube-api-access-d9gxq\") pod \"distinct-w5-da76a2-kserve-6f7448ff85-zmfvb\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") " pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.324161 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.324121    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:46.632854 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:46.632816    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"]
Oct 30 00:54:46.633604 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:54:46.633581    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ad4cd2d_6793_4856_826b_bcb46811db0d.slice/crio-fd9d8277a1d286ddf2fe97d1ea4c89258e20460a9bec135e331a544eac548207 WatchSource:0}: Error finding container fd9d8277a1d286ddf2fe97d1ea4c89258e20460a9bec135e331a544eac548207: Status 404 returned error can't find the container with id fd9d8277a1d286ddf2fe97d1ea4c89258e20460a9bec135e331a544eac548207
Oct 30 00:54:47.593069 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:47.593034    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:47.639089 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:47.639039    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" event={"ID":"6ad4cd2d-6793-4856-826b-bcb46811db0d","Type":"ContainerStarted","Data":"f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a"}
Oct 30 00:54:47.639315 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:47.639113    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" event={"ID":"6ad4cd2d-6793-4856-826b-bcb46811db0d","Type":"ContainerStarted","Data":"fd9d8277a1d286ddf2fe97d1ea4c89258e20460a9bec135e331a544eac548207"}
Oct 30 00:54:47.681843 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:47.681793    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podStartSLOduration=2.681778914 podStartE2EDuration="2.681778914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:47.680476922 +0000 UTC m=+967.751800771" watchObservedRunningTime="2026-10-30 00:54:47.681778914 +0000 UTC m=+967.753102792"
Oct 30 00:54:48.324725 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:48.324687    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:48.381930 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:54:48.381896    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:54:49.239292 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:49.239224    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:50.985119 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:50.985084    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:52.555943 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:52.555903    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:53.138944 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.138912    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"]
Oct 30 00:54:53.142975 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.142955    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.180612 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.176760    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"distinct-2-w6-5a344d-kserve-self-signed-certs\""
Oct 30 00:54:53.180612 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.176776    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"openshift-service-ca.crt\""
Oct 30 00:54:53.180612 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.176774    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"kube-root-ca.crt\""
Oct 30 00:54:53.180612 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.176775    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5a344d\"/\"default-dockercfg-kdmj6\""
Oct 30 00:54:53.180944 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.180751    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"]
Oct 30 00:54:53.222166 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.222137    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:53.237468 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.237439    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-dshm\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.237629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.237497    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-home\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.237629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.237611    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-tmp-dir\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.237756 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.237649    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea6a488-e59a-4891-8c76-1e12758d30b0-tls-certs\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.237756 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.237719    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686vf\" (UniqueName: \"kubernetes.io/projected/2ea6a488-e59a-4891-8c76-1e12758d30b0-kube-api-access-686vf\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.237865 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.237814    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-model-cache\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.338888 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.338854    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-tmp-dir\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.338888 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.338891    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea6a488-e59a-4891-8c76-1e12758d30b0-tls-certs\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.339180 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.338929    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686vf\" (UniqueName: \"kubernetes.io/projected/2ea6a488-e59a-4891-8c76-1e12758d30b0-kube-api-access-686vf\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.339180 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.338980    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-model-cache\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.339180 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.339086    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-dshm\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.339180 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.339139    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-home\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.339460 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.339365    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-tmp-dir\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.339517 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.339457    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-model-cache\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.339639 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.339601    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-home\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.342372 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.342346    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea6a488-e59a-4891-8c76-1e12758d30b0-tls-certs\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.344534 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.344509    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-dshm\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.360949 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.360924    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686vf\" (UniqueName: \"kubernetes.io/projected/2ea6a488-e59a-4891-8c76-1e12758d30b0-kube-api-access-686vf\") pod \"distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") " pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.467114 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.467079    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:53.784509 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:53.784485    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"]
Oct 30 00:54:53.786450 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:54:53.786426    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea6a488_e59a_4891_8c76_1e12758d30b0.slice/crio-7aa9424af75c0d25d5d1a0e47cd5970b2480053477c7ca5705d70add8b4a833f WatchSource:0}: Error finding container 7aa9424af75c0d25d5d1a0e47cd5970b2480053477c7ca5705d70add8b4a833f: Status 404 returned error can't find the container with id 7aa9424af75c0d25d5d1a0e47cd5970b2480053477c7ca5705d70add8b4a833f
Oct 30 00:54:54.272287 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:54.272223    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:54.395624 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:54.395585    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:54.506354 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:54.506293    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:54.694406 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:54.694372    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" event={"ID":"2ea6a488-e59a-4891-8c76-1e12758d30b0","Type":"ContainerStarted","Data":"7982d29ae505e402a3f49ac1d14718366a3e46b83029ea9e3bc7948f17ba0acb"}
Oct 30 00:54:54.694646 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:54.694421    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" event={"ID":"2ea6a488-e59a-4891-8c76-1e12758d30b0","Type":"ContainerStarted","Data":"7aa9424af75c0d25d5d1a0e47cd5970b2480053477c7ca5705d70add8b4a833f"}
Oct 30 00:54:54.739292 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:54.739227    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podStartSLOduration=1.739213363 podStartE2EDuration="1.739213363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:54.737875754 +0000 UTC m=+974.809199590" watchObservedRunningTime="2026-10-30 00:54:54.739213363 +0000 UTC m=+974.810537198"
Oct 30 00:54:55.468407 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:55.468367    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:54:56.038323 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:56.038290    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:56.325152 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:56.325072    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:56.337877 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:56.337854    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:56.719292 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:56.719264    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:54:57.502012 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:57.501977    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:54:59.158588 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:54:59.158532    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:00.603977 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:00.603940    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:02.069333 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:02.069289    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:03.107815 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:03.107779    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:03.468155 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:03.468125    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:55:03.482326 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:03.482301    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:55:03.752965 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:03.752874    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:03.759722 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:03.759698    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:55:05.293587 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:05.293546    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:06.846370 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:06.846338    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:08.382176 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:08.382139    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:09.960760 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:09.960729    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:11.188085 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:11.188051    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:55:11.399676 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:11.399636    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:13.413259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:13.408185    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:15.784971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:15.784933    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:15.956121 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:15.956090    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"]
Oct 30 00:55:15.961661 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:15.961637    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:15.984157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:15.984127    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"]
Oct 30 00:55:16.093571 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.093491    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2g72\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-kube-api-access-t2g72\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.093571 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.093542    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.093771 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.093660    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74987a39-7f9e-4389-80c4-c166af317c0a-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.093771 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.093710    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.093771 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.093751    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.093922 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.093807    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.093922 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.093911    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.094032 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.093966    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.094087 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.094048    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74987a39-7f9e-4389-80c4-c166af317c0a-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195371 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195334    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195592 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195378    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195592 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195423    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195592 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195449    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74987a39-7f9e-4389-80c4-c166af317c0a-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195592 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195564    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2g72\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-kube-api-access-t2g72\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195821 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195613    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195821 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195678    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74987a39-7f9e-4389-80c4-c166af317c0a-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195821 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195731    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195821 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195777    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.195821 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195809    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.196055 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195850    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.196055 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.195971    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.196124 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.196057    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.196449 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.196429    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74987a39-7f9e-4389-80c4-c166af317c0a-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.198288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.198264    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.198396 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.198290    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74987a39-7f9e-4389-80c4-c166af317c0a-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.216883 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.216857    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.217088 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.217066    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2g72\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-kube-api-access-t2g72\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.277914 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.277866    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:16.593435 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.593408    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"]
Oct 30 00:55:16.595410 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:55:16.595381    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74987a39_7f9e_4389_80c4_c166af317c0a.slice/crio-d8b7580f77604ffc38cdc6392180eda4addf5de4b3785ddf6205f64563e42266 WatchSource:0}: Error finding container d8b7580f77604ffc38cdc6392180eda4addf5de4b3785ddf6205f64563e42266: Status 404 returned error can't find the container with id d8b7580f77604ffc38cdc6392180eda4addf5de4b3785ddf6205f64563e42266
Oct 30 00:55:16.598091 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.598054    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:16.598173 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.598139    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:16.598219 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.598205    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:16.837803 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.837762    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" event={"ID":"74987a39-7f9e-4389-80c4-c166af317c0a","Type":"ContainerStarted","Data":"c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba"}
Oct 30 00:55:16.838329 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.837859    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" event={"ID":"74987a39-7f9e-4389-80c4-c166af317c0a","Type":"ContainerStarted","Data":"d8b7580f77604ffc38cdc6392180eda4addf5de4b3785ddf6205f64563e42266"}
Oct 30 00:55:16.883303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:16.883191    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podStartSLOduration=1.883177136 podStartE2EDuration="1.883177136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:16.880547453 +0000 UTC m=+996.951871345" watchObservedRunningTime="2026-10-30 00:55:16.883177136 +0000 UTC m=+996.954500971"
Oct 30 00:55:17.278833 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:17.278801    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:17.280163 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:17.280132    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:55:17.280303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:17.280181    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:55:17.392665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:17.392627    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:18.280260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:18.279650    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:55:18.280260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:18.279731    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:55:18.410137 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:55:18.410098    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:55:18.915804 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:18.915773    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:19.283260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:19.279719    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:55:19.283260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:19.279779    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:55:19.711018 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:19.710969    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:20.280124 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:20.279671    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:55:20.280124 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:20.279745    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:55:20.636911 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:20.636821    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:21.281263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:21.279416    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:55:21.281263 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:21.279495    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:55:22.158726 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:22.158693    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:22.293433 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:22.293394    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:55:22.293669 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:22.293474    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:55:23.279019 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:23.278982    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:55:23.279475 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:23.279066    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:55:23.664267 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:23.664136    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:24.283550 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:24.283521    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:24.284054 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:24.283922    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:24.284689 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:24.284675    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 00:55:25.149222 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:25.149189    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:26.594164 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:26.594126    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:28.044650 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:28.044618    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:29.460012 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:29.459984    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:29.586389 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:29.586360    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:30.509330 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.509283    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:30.647007 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.646974    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"]
Oct 30 00:55:30.650942 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.650921    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.687599 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.687567    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"]
Oct 30 00:55:30.708970 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.708931    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e33b94-880e-4145-9cf6-97b89a72b5b3-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.709273 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.709214    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsqxp\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-kube-api-access-wsqxp\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.709456 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.709432    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.709561 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.709478    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.709656 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.709635    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.709775 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.709728    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.709775 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.709756    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.709941 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.709778    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.709941 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.709817    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811288 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811183    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811318    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e33b94-880e-4145-9cf6-97b89a72b5b3-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811338    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsqxp\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-kube-api-access-wsqxp\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811357    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811380    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811661 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811479    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811661 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811565    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811661 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811591    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811661 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811615    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.811869 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811851    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.812008 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.811988    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.812074 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.812051    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.812074 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.812063    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.812185 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.812122    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e33b94-880e-4145-9cf6-97b89a72b5b3-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.814035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.814016    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.814115 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.814078    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.839045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.839012    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.839183 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.839128    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsqxp\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-kube-api-access-wsqxp\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:30.860315 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.860280    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"]
Oct 30 00:55:30.865450 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.865427    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:30.891190 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.891155    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"]
Oct 30 00:55:30.972021 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:30.971979    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:31.013550 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.013513    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.013724 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.013563    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnkrv\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-kube-api-access-nnkrv\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.013724 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.013709    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015cb6bc-6824-416b-a693-38f261dc435d-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.013853 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.013776    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.013853 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.013835    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.013853 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.013850    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.014035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.013879    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015cb6bc-6824-416b-a693-38f261dc435d-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.014035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.013969    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.014035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.014003    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.115645    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015cb6bc-6824-416b-a693-38f261dc435d-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.115709    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.115785    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.115810    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.115854    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015cb6bc-6824-416b-a693-38f261dc435d-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.115915    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.115968    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.116098    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.116146    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnkrv\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-kube-api-access-nnkrv\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.116861    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.116926    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015cb6bc-6824-416b-a693-38f261dc435d-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.116931    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.117584    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.117872    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.119302    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.122317 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.120942    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015cb6bc-6824-416b-a693-38f261dc435d-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.149279 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.138695    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnkrv\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-kube-api-access-nnkrv\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.149279 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.140619    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.184313 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.184279    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:31.296754 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:55:31.295671    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e33b94_880e_4145_9cf6_97b89a72b5b3.slice/crio-1fb5bd0c33f07380a2aed5534352d5fbff734675d770f2f360c3592ee42a7ee8 WatchSource:0}: Error finding container 1fb5bd0c33f07380a2aed5534352d5fbff734675d770f2f360c3592ee42a7ee8: Status 404 returned error can't find the container with id 1fb5bd0c33f07380a2aed5534352d5fbff734675d770f2f360c3592ee42a7ee8
Oct 30 00:55:31.299470 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.298218    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:31.299470 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.298338    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:31.299470 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.298368    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:31.312998 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.312965    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"]
Oct 30 00:55:31.356710 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.356682    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:31.504533 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.503849    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"]
Oct 30 00:55:31.507269 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:55:31.506983    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015cb6bc_6824_416b_a693_38f261dc435d.slice/crio-8b90398685e0eab28091f7fb795f4a39a14a3ad1dda031f5196810bd2d5c77fe WatchSource:0}: Error finding container 8b90398685e0eab28091f7fb795f4a39a14a3ad1dda031f5196810bd2d5c77fe: Status 404 returned error can't find the container with id 8b90398685e0eab28091f7fb795f4a39a14a3ad1dda031f5196810bd2d5c77fe
Oct 30 00:55:31.509776 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.509743    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:31.510224 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.509830    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:31.510224 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.509873    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:31.965130 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.963936    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" event={"ID":"015cb6bc-6824-416b-a693-38f261dc435d","Type":"ContainerStarted","Data":"8cce920921efc1f479819442eae58de0d30e5586adc50579b88a510468f31560"}
Oct 30 00:55:31.965130 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.964007    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" event={"ID":"015cb6bc-6824-416b-a693-38f261dc435d","Type":"ContainerStarted","Data":"8b90398685e0eab28091f7fb795f4a39a14a3ad1dda031f5196810bd2d5c77fe"}
Oct 30 00:55:31.967185 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.967155    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" event={"ID":"44e33b94-880e-4145-9cf6-97b89a72b5b3","Type":"ContainerStarted","Data":"a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64"}
Oct 30 00:55:31.967426 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.967403    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" event={"ID":"44e33b94-880e-4145-9cf6-97b89a72b5b3","Type":"ContainerStarted","Data":"1fb5bd0c33f07380a2aed5534352d5fbff734675d770f2f360c3592ee42a7ee8"}
Oct 30 00:55:31.972569 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:31.972541    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:32.063553 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:32.063478    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podStartSLOduration=2.063459425 podStartE2EDuration="2.063459425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:32.015633333 +0000 UTC m=+1012.086957195" watchObservedRunningTime="2026-10-30 00:55:32.063459425 +0000 UTC m=+1012.134783261"
Oct 30 00:55:32.066899 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:32.066837    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" podStartSLOduration=2.066825378 podStartE2EDuration="2.066825378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:32.062690931 +0000 UTC m=+1012.134014851" watchObservedRunningTime="2026-10-30 00:55:32.066825378 +0000 UTC m=+1012.138149212"
Oct 30 00:55:32.188254 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:32.187499    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:32.188254 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:32.187636    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:55:32.188254 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:32.187683    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:55:32.685289 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:32.659440    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:33.002258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:33.001407    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:55:33.002258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:33.001484    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 00:55:33.189732 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:33.189696    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:55:33.190002 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:33.189975    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:55:33.536996 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:33.536967    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:33.976154 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:33.975696    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:55:33.976154 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:33.975757    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 00:55:34.188257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:34.186637    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:55:34.188257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:34.186698    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:55:34.979261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:34.976652    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:55:34.979261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:34.976714    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 00:55:35.190257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:35.189625    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:55:35.190257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:35.189691    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:55:35.705895 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:35.705858    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:35.975316 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:35.972992    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:55:35.975316 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:35.973053    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 00:55:36.191260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:36.190670    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:55:36.191260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:36.190733    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:55:37.012259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:36.990559    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:55:37.012259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:36.990638    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 00:55:37.189258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:37.188657    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:55:37.189258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:37.188730    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:55:37.429318 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:37.428454    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:37.973453 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:37.973404    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 00:55:37.973610 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:37.973478    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 00:55:38.185200 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:38.185162    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:55:38.185413 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:38.185222    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:55:39.121775 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:39.121741    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:39.165071 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:39.165031    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:39.166682 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:39.166554    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:39.167547 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:39.167527    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 00:55:39.345651 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:39.345616    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:39.346341 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:39.346305    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:39.347403 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:39.347380    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 00:55:40.942881 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:40.942849    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:42.499087 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:42.499048    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:43.815686 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:43.815654    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:55:44.018401 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:44.018366    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:45.943257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:45.942853    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:45.955256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:45.952874    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"]
Oct 30 00:55:45.974489 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:45.974453    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"]
Oct 30 00:55:45.974800 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:45.974784    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.078005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.077957    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.418052 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.417961    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aebd8f6-9479-449c-9cb4-364cd0737de0-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.418427 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.418400    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.456257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.419656    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.458821 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.458774    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.459119 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.459096    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.460361 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.460340    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.460666 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.460626    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvct\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-kube-api-access-5cvct\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.460970 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.460907    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.568055 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.568010    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.569741 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.569713    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.570361 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.570338    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.570916 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.570898    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.571442 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.571422    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.571957 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.571936    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvct\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-kube-api-access-5cvct\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.576266 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.572408    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.576758 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.576740    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.577214 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.577195    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aebd8f6-9479-449c-9cb4-364cd0737de0-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.616351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.570832    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-credential-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.616351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.571820    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-data\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.616351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.577141    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-certs\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.616351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.571327    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-socket\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.616351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.578001    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aebd8f6-9479-449c-9cb4-364cd0737de0-istiod-ca-cert\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.616351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.611856    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-podinfo\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.617783 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.617756    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvct\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-kube-api-access-5cvct\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.628279 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.626975    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-envoy\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.659693 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.658834    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-token\") pod \"e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") " pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:46.709700 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:46.709444    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:47.264776 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:47.264742    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"]
Oct 30 00:55:47.266564 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:55:47.266536    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aebd8f6_9479_449c_9cb4_364cd0737de0.slice/crio-5463670448a1e8742717fff5ec6ddb9eabb915c2adc20451e84d57c830e47b8f WatchSource:0}: Error finding container 5463670448a1e8742717fff5ec6ddb9eabb915c2adc20451e84d57c830e47b8f: Status 404 returned error can't find the container with id 5463670448a1e8742717fff5ec6ddb9eabb915c2adc20451e84d57c830e47b8f
Oct 30 00:55:47.269456 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:47.269416    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:47.269541 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:47.269511    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:47.269579 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:47.269558    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:55:47.329005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:47.328976    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" event={"ID":"6aebd8f6-9479-449c-9cb4-364cd0737de0","Type":"ContainerStarted","Data":"5463670448a1e8742717fff5ec6ddb9eabb915c2adc20451e84d57c830e47b8f"}
Oct 30 00:55:48.335788 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:48.335753    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" event={"ID":"6aebd8f6-9479-449c-9cb4-364cd0737de0","Type":"ContainerStarted","Data":"11ab48e2a5466fc0097284856a5216c50c3f0284db3619372d9d994df4b77eda"}
Oct 30 00:55:48.393653 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:48.373170    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:48.412187 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:55:48.412125    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:55:48.712498 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:48.712466    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:48.714246 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:48.714206    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:55:48.714390 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:48.714290    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:55:49.713187 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:49.713154    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:55:49.713627 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:49.713204    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:55:50.096741 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:50.096663    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:50.713574 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:50.713541    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:55:50.713969 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:50.713589    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:55:51.713610 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:51.713533    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:55:51.713610 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:51.713595    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:55:51.758968 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:51.758909    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:52.712633 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:52.712601    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:55:52.712846 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:52.712670    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:55:53.397773 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:53.397734    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:53.713112 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:53.713080    4576 patch_prober.go:29] interesting pod/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:55:53.713312 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:53.713147    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:55:54.959169 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:54.959108    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:54.978120 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:54.978088    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:54.978659 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:54.978623    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:54.979993 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:54.979974    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 00:55:55.053106 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:55.053056    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" podStartSLOduration=10.053042629 podStartE2EDuration="10.053042629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:48.409091599 +0000 UTC m=+1028.480415471" watchObservedRunningTime="2026-10-30 00:55:55.053042629 +0000 UTC m=+1035.124366465"
Oct 30 00:55:56.666916 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:56.666869    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:58.248784 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:58.248751    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:58.908366 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:58.908332    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:55:59.710908 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:55:59.710866    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:01.289643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:01.289612    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:01.482297 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:01.482221    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:02.910358 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:02.910323    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:04.569319 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:04.569271    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:06.127655 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:06.127621    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:08.458384 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:08.458341    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:08.499008 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:08.498968    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 00:56:08.543602 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:08.542365    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:08.595386 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:08.595340    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 00:56:08.641460 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:08.641398    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:08.708500 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:08.708401    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:56:09.289295 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:09.289160    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:11.380984 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:11.380936    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:12.923811 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:12.923774    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:14.490523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:14.490492    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:16.149665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:16.149628    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:17.734385 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:17.734344    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:18.381318 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:56:18.381275    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:56:19.507370 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:19.507329    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:21.042035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:21.042002    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:22.570525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:22.570497    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:24.160593 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:24.160558    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:25.241668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:25.241634    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:56:25.738855 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:25.738815    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:28.418380 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:28.418330    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:56:28.466389 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:28.466338    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:28.474400 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:28.474356    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:28.531382 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:28.531330    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:28.716410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:28.716361    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:28.751401 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:28.750348    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:56:29.340259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.325373    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:29.437123 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.437066    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:29.463773 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.463730    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:56:29.498271 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.490319    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 00:56:29.506516 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.506485    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:56:29.507338 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.507300    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:29.529264 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.507570    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 00:56:29.585266 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.523456    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:56:29.585266 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.567617    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 30 00:56:29.585266 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.567657    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:29.585266 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.571291    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:56:29.771049 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.737668    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:56:29.771049 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.738182    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:56:29.771049 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.740556    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 00:56:29.771764 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.771727    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:29.855746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.855709    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:56:29.862825 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:29.856980    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:56:30.382567 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:30.382531    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:56:30.382827 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:30.382814    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:56:30.382938 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:30.382929    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:56:30.383035 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:30.383027    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:56:30.383137 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:30.383128    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:56:30.383313 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:30.383222    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:56:30.856489 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:30.856450    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:31.296764 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:31.296737    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:32.902759 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:32.902716    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:33.477956 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:33.477912    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:35.251117 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:35.251073    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:36.889957 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:36.889899    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:38.681841 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:38.681794    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:40.370181 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:40.370149    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:41.845813 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:41.845782    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:43.449449 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:43.449414    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:45.067662 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.067624    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:45.852509 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.852465    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"]
Oct 30 00:56:45.857123 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.857099    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.894918 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.894882    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"]
Oct 30 00:56:45.952558 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952521    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.952708 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952580    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.952708 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952682    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prnzg\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-kube-api-access-prnzg\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.952807 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952725    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.952807 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952744    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.952871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952844    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.952913 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952900    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.952961 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952946    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:45.952998 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:45.952984    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054384 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054344    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054398    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054427    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prnzg\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-kube-api-access-prnzg\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054463    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054478    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054495    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054543    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054881 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054601    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054881 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054626    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.054881 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.054836    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-credential-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.055039 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.055008    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-data\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.055087 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.055063    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-socket\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.055087 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.055070    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-certs\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.055492 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.055476    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istiod-ca-cert\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.057082 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.057060    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-envoy\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.057219 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.057200    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-podinfo\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.075138 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.075109    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-token\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.075606 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.075164    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prnzg\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-kube-api-access-prnzg\") pod \"e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") " pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.172399 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.172322    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:46.204351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.204322    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"]
Oct 30 00:56:46.209362 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.209336    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.233553 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.232483    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"]
Oct 30 00:56:46.359735 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.359702    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kssfg\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-kube-api-access-kssfg\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.359928 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.359757    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.359928 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.359851    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.360044 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.359972    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.360104 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.360057    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.360104 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.360098    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.360194 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.360164    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.360287 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.360272    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.360333 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.360319    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462008 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.461974    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kssfg\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-kube-api-access-kssfg\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462175 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462031    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462175 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462060    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462175 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462097    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462175 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462152    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462417 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462182    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462417 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462216    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462417 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462312    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462417 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462394    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462625 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462511    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-credential-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462625 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462566    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-certs\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462734 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462617    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-data\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.462787 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462770    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-socket\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.463012 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.462995    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istiod-ca-cert\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.464745 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.464726    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-envoy\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.464966 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.464949    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-podinfo\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.482719 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.482691    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-token\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.482835 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.482761    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kssfg\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-kube-api-access-kssfg\") pod \"e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") " pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.504533 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.504512    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"]
Oct 30 00:56:46.506072 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:56:46.506052    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc5c913_545b_43c0_ac8d_ab314d7ffb77.slice/crio-097144491ed6305cc0246c1596a57576fb20d0311ca74656aa7fd8ae4decb7fb WatchSource:0}: Error finding container 097144491ed6305cc0246c1596a57576fb20d0311ca74656aa7fd8ae4decb7fb: Status 404 returned error can't find the container with id 097144491ed6305cc0246c1596a57576fb20d0311ca74656aa7fd8ae4decb7fb
Oct 30 00:56:46.508522 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.508495    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:56:46.508591 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.508556    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:56:46.508632 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.508588    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:56:46.532501 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.532477    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:46.735885 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.735801    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:46.854358 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.854329    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"]
Oct 30 00:56:46.855593 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:56:46.855568    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3964518_d1f8_4c0b_ab7f_2ff3afec1e15.slice/crio-a275089fd437b52cdf4696c929833ef1c96480eae8a6981fb531eef390a9f8ee WatchSource:0}: Error finding container a275089fd437b52cdf4696c929833ef1c96480eae8a6981fb531eef390a9f8ee: Status 404 returned error can't find the container with id a275089fd437b52cdf4696c929833ef1c96480eae8a6981fb531eef390a9f8ee
Oct 30 00:56:46.858345 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.858314    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:56:46.858438 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.858380    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:56:46.858438 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.858407    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:56:46.932199 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.931320    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" event={"ID":"dbc5c913-545b-43c0-ac8d-ab314d7ffb77","Type":"ContainerStarted","Data":"b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25"}
Oct 30 00:56:46.932199 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.931379    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" event={"ID":"dbc5c913-545b-43c0-ac8d-ab314d7ffb77","Type":"ContainerStarted","Data":"097144491ed6305cc0246c1596a57576fb20d0311ca74656aa7fd8ae4decb7fb"}
Oct 30 00:56:46.933446 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.933413    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" event={"ID":"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15","Type":"ContainerStarted","Data":"a275089fd437b52cdf4696c929833ef1c96480eae8a6981fb531eef390a9f8ee"}
Oct 30 00:56:46.981640 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:46.981567    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podStartSLOduration=1.9815452599999999 podStartE2EDuration="1.98154526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:46.977440372 +0000 UTC m=+1087.048764212" watchObservedRunningTime="2026-10-30 00:56:46.98154526 +0000 UTC m=+1087.052869094"
Oct 30 00:56:47.172858 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:47.172769    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:47.174502 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:47.174478    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:47.174665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:47.174538    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:47.941213 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:47.941174    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" event={"ID":"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15","Type":"ContainerStarted","Data":"dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15"}
Oct 30 00:56:48.018258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:48.016473    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podStartSLOduration=2.016448995 podStartE2EDuration="2.016448995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:47.989764545 +0000 UTC m=+1088.061088382" watchObservedRunningTime="2026-10-30 00:56:48.016448995 +0000 UTC m=+1088.087772830"
Oct 30 00:56:48.200303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:48.195357    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:48.200303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:48.195420    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:48.427179 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:56:48.427136    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:56:48.534012 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:48.533927    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:48.535597 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:48.535570    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 00:56:48.535711 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:48.535650    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 00:56:49.180262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:49.176621    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:49.180262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:49.176680    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:49.547261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:49.542373    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 00:56:49.547261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:49.542434    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 00:56:50.182262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.181406    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:50.182498 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.182407    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:50.182498 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.182446    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:50.216260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.215373    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:50.241261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.240326    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:50.281830 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.281790    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:50.290639 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.290601    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:50.308072 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.306584    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:50.308072 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.307398    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:50.697983 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.639310    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 00:56:50.704521 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.704480    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 00:56:50.747014 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.718580    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 00:56:50.749688 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.749653    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:50.765045 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.765011    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 00:56:50.816211 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.812341    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:50.817492 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:50.817457    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 00:56:51.238322 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.210035    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Oct 30 00:56:51.238322 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.218802    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:56:51.238322 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.220018    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:56:51.238322 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.223176    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:51.238322 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.223626    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:51.297260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.249266    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:51.297260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.249738    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:56:51.336260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.317912    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:51.336260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.334093    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:51.336260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.334304    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:51.364188 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.364155    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:56:51.414381 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.414336    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 00:56:51.433261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.416098    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:56:51.433637 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.433613    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:56:51.597324 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.594072    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:56:51.668260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.652351    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 00:56:51.668260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.652409    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 00:56:51.700257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.674859    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:51.701615 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.701587    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:56:51.763170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.763121    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:51.771069 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.769359    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:56:51.788583 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.776015    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:51.817429 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.817388    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:51.826298 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.821993    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:56:51.942333 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:51.933068    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:56:52.016126 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.016099    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:56:52.179887 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.179769    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:52.180319 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.180287    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:52.227712 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.223394    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:52.255932 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.255892    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 00:56:52.374257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.370395    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:52.419418 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.419372    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:52.429369 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.429331    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:52.559422 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.547278    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 00:56:52.559422 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.547342    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 00:56:52.728258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.703117    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 00:56:52.787264 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.780086    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 00:56:52.805260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:52.799442    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 00:56:53.184260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.174816    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:53.184260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.174877    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:53.229335 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.229041    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:53.326283 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.304454    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 00:56:53.397259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.396330    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:53.456320 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.424492    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 00:56:53.481289 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.451637    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:53.548468 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.548428    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 00:56:53.548745 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.548715    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 00:56:53.554438 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.554412    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:56:53.603536 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.603499    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:56:53.661339 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.661296    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:53.771322 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.765402    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:53.818262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.817416    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:53.845679 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.844726    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:56:53.848387 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.848357    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:53.848910 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.848893    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:56:53.857186 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.857168    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:56:53.866014 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.865995    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:56:53.870175 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.869586    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:56:53.899644 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.874209    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:56:53.923617 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:53.923592    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:56:54.175424 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:54.175339    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:54.203380 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:54.203334    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:54.533588 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:54.533557    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 00:56:54.533771 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:54.533607    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 00:56:55.122977 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:55.122941    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:55.197256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:55.183709    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:56:55.197256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:55.183770    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:56:55.533379 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:55.533341    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 00:56:55.533569 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:55.533396    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 00:56:57.214441 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:57.214400    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:56:57.214959 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:57.214467    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:57.575546 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:57.575437    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:56:57.575546 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:57.575528    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:56:58.002629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:58.002589    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:58.003458 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:58.003414    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:58.004418 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:58.004392    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 00:56:58.105708 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:58.105675    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:56:58.110287 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:58.110261    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:58.110658 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:58.110635    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:58.111961 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:58.111939    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 00:56:59.655222 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.655191    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"]
Oct 30 00:56:59.658635 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.658617    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.677206 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.677177    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"]
Oct 30 00:56:59.686803 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.686775    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:56:59.686803 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.686781    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 00:56:59.733659 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.733627    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.733659 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.733660    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ht4\" (UniqueName: \"kubernetes.io/projected/65d8a748-02fc-46c2-8b88-d67ffce24f16-kube-api-access-h7ht4\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.733963 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.733939    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.835741 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.835708    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.835907 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.835804    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.835907 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.835823    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ht4\" (UniqueName: \"kubernetes.io/projected/65d8a748-02fc-46c2-8b88-d67ffce24f16-kube-api-access-h7ht4\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.836498 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.836465    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.838397 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.838376    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.856499 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.856468    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ht4\" (UniqueName: \"kubernetes.io/projected/65d8a748-02fc-46c2-8b88-d67ffce24f16-kube-api-access-h7ht4\") pod \"maas-api-revoke-sub-2eaca62e-sshc9\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:56:59.973436 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:56:59.973402    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:57:00.095978 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:00.095944    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:00.279197 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:00.276455    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"]
Oct 30 00:57:00.286586 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:57:00.286559    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d8a748_02fc_46c2_8b88_d67ffce24f16.slice/crio-2fca11b7f21b47a9ecf1738d0125e033fb1286fa0d926cbda6299ee1b42fd958 WatchSource:0}: Error finding container 2fca11b7f21b47a9ecf1738d0125e033fb1286fa0d926cbda6299ee1b42fd958: Status 404 returned error can't find the container with id 2fca11b7f21b47a9ecf1738d0125e033fb1286fa0d926cbda6299ee1b42fd958
Oct 30 00:57:00.901334 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:00.901213    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9" event={"ID":"65d8a748-02fc-46c2-8b88-d67ffce24f16","Type":"ContainerStarted","Data":"8d8b5b65f5ff5d1f59c0f02c9d4658c87634a411b9267966c5d647947e24236c"}
Oct 30 00:57:00.901334 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:00.901299    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9" event={"ID":"65d8a748-02fc-46c2-8b88-d67ffce24f16","Type":"ContainerStarted","Data":"2fca11b7f21b47a9ecf1738d0125e033fb1286fa0d926cbda6299ee1b42fd958"}
Oct 30 00:57:00.943110 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:00.943048    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9" podStartSLOduration=1.943030362 podStartE2EDuration="1.943030362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:00.942420791 +0000 UTC m=+1101.013744627" watchObservedRunningTime="2026-10-30 00:57:00.943030362 +0000 UTC m=+1101.014354199"
Oct 30 00:57:01.908712 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:01.908680    4576 generic.go:356] "Generic (PLEG): container finished" podID="65d8a748-02fc-46c2-8b88-d67ffce24f16" containerID="8d8b5b65f5ff5d1f59c0f02c9d4658c87634a411b9267966c5d647947e24236c" exitCode=0
Oct 30 00:57:01.909168 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:01.908768    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9" event={"ID":"65d8a748-02fc-46c2-8b88-d67ffce24f16","Type":"ContainerDied","Data":"8d8b5b65f5ff5d1f59c0f02c9d4658c87634a411b9267966c5d647947e24236c"}
Oct 30 00:57:01.927682 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:01.927649    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:02.675700 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:02.675655    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:57:03.103159 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.103129    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:57:03.180427 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.180393    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-openshift-ca\") pod \"65d8a748-02fc-46c2-8b88-d67ffce24f16\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") "
Oct 30 00:57:03.180604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.180525    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-xks-ca\") pod \"65d8a748-02fc-46c2-8b88-d67ffce24f16\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") "
Oct 30 00:57:03.180604 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.180568    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d8a748-02fc-46c2-8b88-d67ffce24f16-kube-api-access-h7ht4\" (UniqueName: \"kubernetes.io/projected/65d8a748-02fc-46c2-8b88-d67ffce24f16-kube-api-access-h7ht4\") pod \"65d8a748-02fc-46c2-8b88-d67ffce24f16\" (UID: \"65d8a748-02fc-46c2-8b88-d67ffce24f16\") "
Oct 30 00:57:03.180858 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.180808    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-openshift-ca" pod "65d8a748-02fc-46c2-8b88-d67ffce24f16" (UID: "65d8a748-02fc-46c2-8b88-d67ffce24f16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:03.181376 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.181355    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:03.182796 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.182770    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-xks-ca" pod "65d8a748-02fc-46c2-8b88-d67ffce24f16" (UID: "65d8a748-02fc-46c2-8b88-d67ffce24f16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:03.183433 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.183406    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d8a748-02fc-46c2-8b88-d67ffce24f16-kube-api-access-h7ht4" pod "65d8a748-02fc-46c2-8b88-d67ffce24f16" (UID: "65d8a748-02fc-46c2-8b88-d67ffce24f16"). InnerVolumeSpecName "kube-api-access-h7ht4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:03.282545 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.282465    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d8a748-02fc-46c2-8b88-d67ffce24f16-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:03.282545 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:03.282490    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ht4\" (UniqueName: \"kubernetes.io/projected/65d8a748-02fc-46c2-8b88-d67ffce24f16-kube-api-access-h7ht4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:04.014259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:04.014204    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9" event={"ID":"65d8a748-02fc-46c2-8b88-d67ffce24f16","Type":"ContainerDied","Data":"2fca11b7f21b47a9ecf1738d0125e033fb1286fa0d926cbda6299ee1b42fd958"}
Oct 30 00:57:04.014557 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:04.014543    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fca11b7f21b47a9ecf1738d0125e033fb1286fa0d926cbda6299ee1b42fd958"
Oct 30 00:57:04.014709 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:04.014699    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"
Oct 30 00:57:04.046906 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:04.046865    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:05.433459 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:05.433394    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:05.486383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:05.486333    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:05.675378 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:05.675330    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:05.772272 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:05.770378    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:05.876272 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:05.875377    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:05.940375 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:05.940326    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:06.032361 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.031707    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:06.262260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.261381    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:06.435935 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.435832    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 00:57:06.437486 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.437456    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:57:06.795256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.785602    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:06.795256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.786157    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 00:57:06.795256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.790966    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:57:06.983261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.892091    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:06.983261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.892916    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:57:07.009258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:06.990326    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:07.038525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.038450    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:07.040013 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.039976    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:57:07.059577 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.059480    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:57:07.504671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.502400    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 00:57:07.504671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.502792    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:57:07.504671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.503028    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:07.504671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.503219    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 30 00:57:07.504671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.503810    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:57:07.504671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.504352    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 00:57:07.790549 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.790418    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:07.794895 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.794859    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 00:57:07.814918 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.813154    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:57:07.910451 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.910397    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:57:07.988472 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.988413    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 00:57:07.990539 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:07.990506    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:08.058638 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.058537    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 00:57:08.492285 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.468133    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:08.507460 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.507417    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:08.517818 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.517766    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:08.518566 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.517774    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 00:57:08.586762 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.586720    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:08.688286 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.614008    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:57:08.688286 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.614071    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:57:08.736270 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.731539    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:57:08.754167 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.754096    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:57:08.765249 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.757080    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:57:08.782344 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.782320    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:57:08.784092 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.784067    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:57:08.805580 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.793348    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:08.819257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.812082    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:57:08.819257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.814844    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:57:08.872059 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.872031    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:57:08.872349 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:08.872333    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:57:11.522324 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:11.522283    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:13.154856 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:13.154822    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:14.708884 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:14.708849    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:16.268350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:16.268305    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:18.523478 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.512416    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.591098 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.591053    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 00:57:18.672144 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.672004    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:18.705081 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:57:18.705038    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:57:18.767396 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.767350    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.767941 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.767910    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:57:18.790326 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.781911    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.831717 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.831669    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:57:18.832220 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.832193    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 00:57:18.861327 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.846439    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:18.970502 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.970453    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 00:57:18.972091 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:18.972057    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:21.251836 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:21.251803    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:22.944555 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:22.944521    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:24.668643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:24.668606    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:26.207854 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:26.207815    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:27.713132 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:27.713098    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:29.253402 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.253370    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:29.823133 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.823100    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"]
Oct 30 00:57:29.823921 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.823895    4576 cpu_manager.go:414] "containerMap: removing container" podUID="65d8a748-02fc-46c2-8b88-d67ffce24f16" containerName="revoke-keys"
Oct 30 00:57:29.823921 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.823916    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d8a748-02fc-46c2-8b88-d67ffce24f16" containerName="revoke-keys"
Oct 30 00:57:29.824084 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.824071    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d8a748-02fc-46c2-8b88-d67ffce24f16" containerName="revoke-keys"
Oct 30 00:57:29.827879 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.827858    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:29.845829 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.845803    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"]
Oct 30 00:57:29.852517 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.852493    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:57:29.852664 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.852645    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 00:57:29.930097 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.930048    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:29.930278 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.930119    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng4p\" (UniqueName: \"kubernetes.io/projected/e99256b7-7ba1-4efe-8fc0-434e0481b30d-kube-api-access-xng4p\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:29.930331 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:29.930281    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:30.031883 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.031848    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:30.032095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.031903    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng4p\" (UniqueName: \"kubernetes.io/projected/e99256b7-7ba1-4efe-8fc0-434e0481b30d-kube-api-access-xng4p\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:30.032095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.031982    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:30.032555 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.032532    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:30.034538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.034515    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:30.053170 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.053142    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng4p\" (UniqueName: \"kubernetes.io/projected/e99256b7-7ba1-4efe-8fc0-434e0481b30d-kube-api-access-xng4p\") pod \"maas-api-revoke-sub-0e51eefe-89t4h\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:30.142587 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.142512    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:30.439563 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.439533    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"]
Oct 30 00:57:30.440888 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:57:30.440852    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99256b7_7ba1_4efe_8fc0_434e0481b30d.slice/crio-545d7740dd3571c0997cfa436ea6dc374d378573d3f09cb833ca117897ba3ea5 WatchSource:0}: Error finding container 545d7740dd3571c0997cfa436ea6dc374d378573d3f09cb833ca117897ba3ea5: Status 404 returned error can't find the container with id 545d7740dd3571c0997cfa436ea6dc374d378573d3f09cb833ca117897ba3ea5
Oct 30 00:57:30.746464 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:30.746355    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:31.050204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:31.050128    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:31.122707 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:31.122673    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:57:31.242094 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:31.242046    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h" event={"ID":"e99256b7-7ba1-4efe-8fc0-434e0481b30d","Type":"ContainerStarted","Data":"9ca6ef38f2bcf790be3e3b0940420f1365d33c448a092726c40f944bc45ae55c"}
Oct 30 00:57:31.242321 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:31.242120    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h" event={"ID":"e99256b7-7ba1-4efe-8fc0-434e0481b30d","Type":"ContainerStarted","Data":"545d7740dd3571c0997cfa436ea6dc374d378573d3f09cb833ca117897ba3ea5"}
Oct 30 00:57:31.285541 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:31.285493    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h" podStartSLOduration=2.285478535 podStartE2EDuration="2.285478535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:31.282944886 +0000 UTC m=+1131.354268749" watchObservedRunningTime="2026-10-30 00:57:31.285478535 +0000 UTC m=+1131.356802369"
Oct 30 00:57:32.254020 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:32.253980    4576 generic.go:356] "Generic (PLEG): container finished" podID="e99256b7-7ba1-4efe-8fc0-434e0481b30d" containerID="9ca6ef38f2bcf790be3e3b0940420f1365d33c448a092726c40f944bc45ae55c" exitCode=0
Oct 30 00:57:32.254806 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:32.254777    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h" event={"ID":"e99256b7-7ba1-4efe-8fc0-434e0481b30d","Type":"ContainerDied","Data":"9ca6ef38f2bcf790be3e3b0940420f1365d33c448a092726c40f944bc45ae55c"}
Oct 30 00:57:33.399260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.398672    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:33.517268 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.508653    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:33.637009 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.636902    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:33.760393 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.760344    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:33.762610 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.761046    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:33.795370 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.795323    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:33.815381 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.815334    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:33.816541 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.816506    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:33.875052 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:33.875009    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:34.073271 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.069900    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 00:57:34.103264 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.097448    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": context deadline exceeded"
Oct 30 00:57:34.610207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.610166    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 30 00:57:34.611274 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.611218    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:34.627272 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.626581    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 00:57:34.785546 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.785501    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:34.798262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.787145    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:57:34.800002 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.799973    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:57:34.801371 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.789342    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:34.822010 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.821984    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:57:34.823332 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.823287    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:34.823506 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.820749    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:34.824657 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.801877    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:34.863027 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.862938    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:57:34.889321 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.887738    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:34.932538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.889350    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:57:34.964261 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.941497    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:57:34.976259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:34.964589    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:57:35.075164 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.075121    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:35.084255 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.079288    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:57:35.418139 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.418109    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:35.533262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.532486    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99256b7-7ba1-4efe-8fc0-434e0481b30d-kube-api-access-xng4p\" (UniqueName: \"kubernetes.io/projected/e99256b7-7ba1-4efe-8fc0-434e0481b30d-kube-api-access-xng4p\") pod \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") "
Oct 30 00:57:35.533262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.532750    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-openshift-ca\") pod \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") "
Oct 30 00:57:35.533262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.532824    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-xks-ca\") pod \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\" (UID: \"e99256b7-7ba1-4efe-8fc0-434e0481b30d\") "
Oct 30 00:57:35.639082 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.627971    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-openshift-ca" pod "e99256b7-7ba1-4efe-8fc0-434e0481b30d" (UID: "e99256b7-7ba1-4efe-8fc0-434e0481b30d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:35.639082 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.629011    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:35.651606 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.651569    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:57:35.664271 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.653876    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 00:57:35.664676 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.664644    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:57:35.684890 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.684798    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:35.724410 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.724377    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.800817 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.800774    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99256b7-7ba1-4efe-8fc0-434e0481b30d-kube-api-access-xng4p" pod "e99256b7-7ba1-4efe-8fc0-434e0481b30d" (UID: "e99256b7-7ba1-4efe-8fc0-434e0481b30d"). InnerVolumeSpecName "kube-api-access-xng4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:35.802453 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.801085    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-xks-ca" pod "e99256b7-7ba1-4efe-8fc0-434e0481b30d" (UID: "e99256b7-7ba1-4efe-8fc0-434e0481b30d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:35.802624 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.802294    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:57:35.824740 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.802125    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 00:57:35.825058 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.825030    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:35.825461 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.824329    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:35.825891 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.825873    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:57:35.827017 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.824368    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:35.880083 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.880046    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:57:35.914304 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.910249    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xng4p\" (UniqueName: \"kubernetes.io/projected/e99256b7-7ba1-4efe-8fc0-434e0481b30d-kube-api-access-xng4p\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.914559 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.914542    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99256b7-7ba1-4efe-8fc0-434e0481b30d-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:35.966538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.947502    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 00:57:35.969059 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.969021    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h" event={"ID":"e99256b7-7ba1-4efe-8fc0-434e0481b30d","Type":"ContainerDied","Data":"545d7740dd3571c0997cfa436ea6dc374d378573d3f09cb833ca117897ba3ea5"}
Oct 30 00:57:35.969325 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.969299    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="545d7740dd3571c0997cfa436ea6dc374d378573d3f09cb833ca117897ba3ea5"
Oct 30 00:57:35.969512 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:35.969498    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"
Oct 30 00:57:36.226306 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.223654    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:36.709737 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.705421    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 00:57:36.732221 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.712976    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 00:57:36.785505 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.778412    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:57:36.801350 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.800599    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:57:36.813314 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.802334    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:57:36.837064 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.831515    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:36.837064 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.831860    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:57:36.837496 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.837454    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 00:57:36.902444 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.900292    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:36.903149 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.903037    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:57:36.906333 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:36.904349    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:57:37.014617 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:37.013590    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:57:37.016025 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:37.015035    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 00:57:37.133144 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:37.129961    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:57:37.140883 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:37.137974    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:57:37.146875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:37.146850    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:57:39.979548 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:39.979509    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:40.275804 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.275723    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"]
Oct 30 00:57:40.276358 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.276340    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e99256b7-7ba1-4efe-8fc0-434e0481b30d" containerName="revoke-keys"
Oct 30 00:57:40.276358 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.276357    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99256b7-7ba1-4efe-8fc0-434e0481b30d" containerName="revoke-keys"
Oct 30 00:57:40.276490 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.276454    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99256b7-7ba1-4efe-8fc0-434e0481b30d" containerName="revoke-keys"
Oct 30 00:57:40.280726 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.280705    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.300699 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.300659    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"]
Oct 30 00:57:40.310598 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.310568    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:57:40.310598 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.310584    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 00:57:40.320866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.320449    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.320866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.320566    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdbs\" (UniqueName: \"kubernetes.io/projected/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-kube-api-access-hzdbs\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.320866 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.320612    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.422741 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.422702    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.422950 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.422758    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdbs\" (UniqueName: \"kubernetes.io/projected/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-kube-api-access-hzdbs\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.422950 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.422779    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.423519 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.423446    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.425477 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.425446    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.446366 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.446329    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdbs\" (UniqueName: \"kubernetes.io/projected/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-kube-api-access-hzdbs\") pod \"maas-api-revoke-sub-009641b3-xfvkq\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.598715 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.598639    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:40.896117 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:40.896087    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"]
Oct 30 00:57:40.897092 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:57:40.897062    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a4a01b_9c4b_4265_a5c0_0fa11d7ccdc8.slice/crio-c1190853e67b5cb36acf97b56693064895ab82fc3ee40268bd61afcefd65a976 WatchSource:0}: Error finding container c1190853e67b5cb36acf97b56693064895ab82fc3ee40268bd61afcefd65a976: Status 404 returned error can't find the container with id c1190853e67b5cb36acf97b56693064895ab82fc3ee40268bd61afcefd65a976
Oct 30 00:57:41.186177 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:41.186147    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq" event={"ID":"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8","Type":"ContainerStarted","Data":"c1190853e67b5cb36acf97b56693064895ab82fc3ee40268bd61afcefd65a976"}
Oct 30 00:57:41.535758 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:41.535716    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:42.193527 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:42.193495    4576 generic.go:356] "Generic (PLEG): container finished" podID="65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8" containerID="bfb067d09dd5d53bb31e7ba5979399eeff4ffe0c363b513c869aac2efc3f7ffd" exitCode=0
Oct 30 00:57:42.193919 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:42.193555    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq" event={"ID":"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8","Type":"ContainerDied","Data":"bfb067d09dd5d53bb31e7ba5979399eeff4ffe0c363b513c869aac2efc3f7ffd"}
Oct 30 00:57:43.159870 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.159836    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:43.375611 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.375580    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:43.457952 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.457921    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-openshift-ca\") pod \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") "
Oct 30 00:57:43.458157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.458053    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-kube-api-access-hzdbs\" (UniqueName: \"kubernetes.io/projected/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-kube-api-access-hzdbs\") pod \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") "
Oct 30 00:57:43.458157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.458108    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-xks-ca\") pod \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\" (UID: \"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8\") "
Oct 30 00:57:43.458385 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.458347    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-openshift-ca" pod "65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8" (UID: "65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:43.458569 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.458542    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:43.460405 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.460376    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-kube-api-access-hzdbs" pod "65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8" (UID: "65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8"). InnerVolumeSpecName "kube-api-access-hzdbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:43.460500 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.460378    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-xks-ca" pod "65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8" (UID: "65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:43.559875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.559828    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:43.559875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.559876    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdbs\" (UniqueName: \"kubernetes.io/projected/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8-kube-api-access-hzdbs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:57:43.603510 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.603478    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:43.737075 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:43.736984    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:44.209017 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:44.208988    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"
Oct 30 00:57:44.209183 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:44.208992    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq" event={"ID":"65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8","Type":"ContainerDied","Data":"c1190853e67b5cb36acf97b56693064895ab82fc3ee40268bd61afcefd65a976"}
Oct 30 00:57:44.209183 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:44.209109    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1190853e67b5cb36acf97b56693064895ab82fc3ee40268bd61afcefd65a976"
Oct 30 00:57:44.248756 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:44.248724    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:44.881323 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:44.881268    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:46.467701 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:46.467664    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:48.378455 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:48.378415    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:48.381513 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:57:48.381484    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:57:50.096002 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:50.095962    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:50.508210 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:50.508175    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:50.624453 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:50.624411    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:50.857468 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:50.857387    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:51.325672 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:51.325637    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:51.455934 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:51.455899    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:51.730573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:51.730538    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:52.131207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.131104    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"]
Oct 30 00:57:52.131891 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.131867    4576 cpu_manager.go:414] "containerMap: removing container" podUID="65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8" containerName="revoke-keys"
Oct 30 00:57:52.132430 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.132412    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8" containerName="revoke-keys"
Oct 30 00:57:52.132623 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.132604    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8" containerName="revoke-keys"
Oct 30 00:57:52.155272 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.149938    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.173408 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.158021    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"]
Oct 30 00:57:52.215915 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.215881    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:52.247245 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.247202    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a03b2130-openshift-default-dockercfg-hsnnd\""
Oct 30 00:57:52.285082 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285040    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-socket\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.285305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285113    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-data\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.285305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285212    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-envoy\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.285447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285347    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9tm8\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-kube-api-access-d9tm8\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.285447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285383    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-istio-token\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.285447 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285418    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-certs\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.285595 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285530    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd54faa-802b-4f59-b38d-58921167451a-istio-podinfo\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.285646 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285632    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd54faa-802b-4f59-b38d-58921167451a-istiod-ca-cert\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.285717 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.285699    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-credential-socket\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387003 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.386911    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd54faa-802b-4f59-b38d-58921167451a-istio-podinfo\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387011    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd54faa-802b-4f59-b38d-58921167451a-istiod-ca-cert\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387071    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-credential-socket\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387114    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-socket\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387156    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-data\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387184    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-envoy\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387275    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9tm8\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-kube-api-access-d9tm8\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387310    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-istio-token\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387343    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-certs\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387943 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387633    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-credential-socket\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387943 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387884    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-data\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387943 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387900    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-socket\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.387943 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.387912    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd54faa-802b-4f59-b38d-58921167451a-istiod-ca-cert\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.388129 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.388053    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-certs\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.389787 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.389766    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd54faa-802b-4f59-b38d-58921167451a-istio-podinfo\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.390157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.390138    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-envoy\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.398817 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.398788    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:52.409305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.409271    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9tm8\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-kube-api-access-d9tm8\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.409483 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.409464    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-istio-token\") pod \"e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") " pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.475510 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.475479    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:52.740834 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.740767    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:52.807038 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.806932    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"]
Oct 30 00:57:52.810992 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:57:52.810959    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd54faa_802b_4f59_b38d_58921167451a.slice/crio-675f3502f74ac81c70caeb98f6dcfdeecf581fc951df24134958139251da7daf WatchSource:0}: Error finding container 675f3502f74ac81c70caeb98f6dcfdeecf581fc951df24134958139251da7daf: Status 404 returned error can't find the container with id 675f3502f74ac81c70caeb98f6dcfdeecf581fc951df24134958139251da7daf
Oct 30 00:57:52.813643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.813608    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:57:52.813753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.813675    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:57:52.813753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:52.813705    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:57:53.270511 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:53.270474    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" event={"ID":"bcd54faa-802b-4f59-b38d-58921167451a","Type":"ContainerStarted","Data":"44b2a496224a459b97731c1cf7a2d9c5b550c3af905ed8b3d8dc3c55fd49263f"}
Oct 30 00:57:53.270722 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:53.270539    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" event={"ID":"bcd54faa-802b-4f59-b38d-58921167451a","Type":"ContainerStarted","Data":"675f3502f74ac81c70caeb98f6dcfdeecf581fc951df24134958139251da7daf"}
Oct 30 00:57:53.322808 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:53.322754    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" podStartSLOduration=1.322737551 podStartE2EDuration="1.322737551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:53.321814858 +0000 UTC m=+1153.393138921" watchObservedRunningTime="2026-10-30 00:57:53.322737551 +0000 UTC m=+1153.394061386"
Oct 30 00:57:53.340492 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:53.340451    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:53.464256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:53.461077    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:53.480265 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:53.478449    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:53.493003 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:53.492976    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:53.582955 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:53.582862    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:54.277565 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:54.277537    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:54.278405 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:54.278381    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:57:54.303139 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:54.303101    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:55.178676 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:55.178635    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:55.295762 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:55.295726    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:55.412949 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:55.410716    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:55.733381 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:55.733339    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:55.869276 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:55.869157    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:57.079256 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:57.078842    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:57.464996 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:57.463037    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:58.439084 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:58.439051    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:57:58.766464 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:58.766351    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.794726 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:58.794651    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.817577 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:58.817537    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.872834 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:58.872784    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.922456 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:58.922399    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.934386 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:58.934348    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:58.939381 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:58.939311    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:59.094214 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.093390    4576 patch_prober.go:29] interesting pod/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:57:59.094717 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.094484    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:59.185470 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.185433    4576 patch_prober.go:29] interesting pod/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:57:59.185753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.185727    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:59.309390 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.309338    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:59.805448 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.798355    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:59.805448 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.800819    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:57:59.847629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.824738    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 00:57:59.847629 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.831313    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:57:59.874949 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.874020    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 00:57:59.874949 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.874477    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:57:59.945513 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.945471    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:59.948758 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.948718    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:59.989500 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.948971    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 00:57:59.990079 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.979793    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:57:59.990820 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.990799    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:57:59.991010 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:57:59.990996    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:58:00.257146 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.257118    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:58:00.311535 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.311506    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:58:00.361273 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.338832    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:58:00.736259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.733808    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:00.736259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.734877    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded"
Oct 30 00:58:00.895260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.869740    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:58:00.923262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.896863    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 00:58:00.946678 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.946636    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:00.947351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.947327    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:58:00.996819 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.996725    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 00:58:00.997426 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.997395    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:00.997920 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:00.997901    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:58:01.213501 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.213452    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:01.273510 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.271599    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:01.333476 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.333430    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:01.446262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.347486    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 00:58:01.738375 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.738329    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 00:58:01.739542 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.739514    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:58:01.905451 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.905394    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 00:58:01.922965 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.922936    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:58:01.958895 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.958844    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 00:58:01.982861 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:01.958704    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:02.013952 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.013486    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:02.014335 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.014293    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 00:58:02.127737 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.127693    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:02.279428 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.279327    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:02.280490 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.280467    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:58:02.398374 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.398334    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 00:58:02.449457 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.449399    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 00:58:02.463130 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.463097    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:58:02.463440 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.463424    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:58:02.761138 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.761090    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 00:58:02.913562 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.913537    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 00:58:02.917031 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.917005    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 00:58:02.993997 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:02.990676    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:03.029383 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.024801    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 00:58:03.071664 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.071534    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:03.079549 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.073014    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:03.080621 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.073944    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 00:58:03.123073 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.106877    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 00:58:03.123073 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.122364    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 00:58:03.125341 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.124312    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 00:58:03.126512 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.126486    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 00:58:03.872269 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:03.872214    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:04.099573 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:04.099516    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:06.560849 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:06.560810    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:09.037369 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:09.037339    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:10.556493 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:10.556456    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:12.096079 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:12.096048    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:13.869161 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:13.869122    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:13.877305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:13.877278    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:58:15.475994 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:15.475959    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:15.603764 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:15.603730    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:17.128260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:17.127638    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:17.926748 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:17.926710    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:18.381421 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:58:18.381380    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:58:19.970915 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:19.970878    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"]
Oct 30 00:58:19.976182 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:19.976162    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:19.994269 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:19.994223    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"]
Oct 30 00:58:20.008512 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.008486    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a03b2130-serving-cert\""
Oct 30 00:58:20.073677 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.073641    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:20.109776 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.109750    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf488d7-8c8b-4902-bb69-60f1be863f13-openshift-service-ca\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.109946 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.109813    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8l4f\" (UniqueName: \"kubernetes.io/projected/baf488d7-8c8b-4902-bb69-60f1be863f13-kube-api-access-d8l4f\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.110018 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.109998    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-metrics-certs\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.110082 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.110065    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-maas-api-tls\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.189581 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.189551    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:20.211190 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.211154    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-metrics-certs\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.211405 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.211222    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-maas-api-tls\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.211405 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.211373    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf488d7-8c8b-4902-bb69-60f1be863f13-openshift-service-ca\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.211533 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.211444    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l4f\" (UniqueName: \"kubernetes.io/projected/baf488d7-8c8b-4902-bb69-60f1be863f13-kube-api-access-d8l4f\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.212267 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.212206    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf488d7-8c8b-4902-bb69-60f1be863f13-openshift-service-ca\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.214032 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.214004    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-maas-api-tls\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.214302 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.214285    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-metrics-certs\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.235791 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.235679    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8l4f\" (UniqueName: \"kubernetes.io/projected/baf488d7-8c8b-4902-bb69-60f1be863f13-kube-api-access-d8l4f\") pod \"maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.296541 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.296512    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:20.302063 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.302029    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:20.609135 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.609107    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"]
Oct 30 00:58:20.610451 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:58:20.610423    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf488d7_8c8b_4902_bb69_60f1be863f13.slice/crio-7daeb18f7a47d21cf2685d5d1117b6d897ea66a97be66166953e12604028e003 WatchSource:0}: Error finding container 7daeb18f7a47d21cf2685d5d1117b6d897ea66a97be66166953e12604028e003: Status 404 returned error can't find the container with id 7daeb18f7a47d21cf2685d5d1117b6d897ea66a97be66166953e12604028e003
Oct 30 00:58:20.611885 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.611868    4576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:58:20.858656 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:20.858538    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:21.477773 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.477740    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh" event={"ID":"baf488d7-8c8b-4902-bb69-60f1be863f13","Type":"ContainerStarted","Data":"bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66"}
Oct 30 00:58:21.478207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.477796    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh" event={"ID":"baf488d7-8c8b-4902-bb69-60f1be863f13","Type":"ContainerStarted","Data":"7daeb18f7a47d21cf2685d5d1117b6d897ea66a97be66166953e12604028e003"}
Oct 30 00:58:21.478207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.477834    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:21.522199 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.522145    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh" podStartSLOduration=2.160485326 podStartE2EDuration="2.522129225s" totalImagesPullingTime="361.643899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:21.519630217 +0000 UTC m=+1181.590954053" watchObservedRunningTime="2026-10-30 00:58:21.522129225 +0000 UTC m=+1181.593453131"
Oct 30 00:58:21.745399 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.745301    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"]
Oct 30 00:58:21.750487 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.750454    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.768466 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.768432    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"]
Oct 30 00:58:21.769351 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.769325    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-baf5781e-serving-cert\""
Oct 30 00:58:21.833698 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.833668    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-metrics-certs\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.833878 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.833720    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.833935 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.833895    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624705a1-b9fe-4606-96fe-f6d993bcd422-openshift-service-ca\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.833986 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.833955    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d622\" (UniqueName: \"kubernetes.io/projected/624705a1-b9fe-4606-96fe-f6d993bcd422-kube-api-access-2d622\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.853646 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.853599    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:21.934907 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.934874    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624705a1-b9fe-4606-96fe-f6d993bcd422-openshift-service-ca\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.935127 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.934959    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d622\" (UniqueName: \"kubernetes.io/projected/624705a1-b9fe-4606-96fe-f6d993bcd422-kube-api-access-2d622\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.935127 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.935099    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-metrics-certs\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.935290 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.935171    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.935405 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:58:21.935386    4576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-serving-cert: secret "maas-api-e2e-ait-baf5781e-serving-cert" not found
Oct 30 00:58:21.935579 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:58:21.935455    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls podName:624705a1-b9fe-4606-96fe-f6d993bcd422 nodeName:}" failed. No retries permitted until 2026-10-30 00:58:22.435436905 +0000 UTC m=+1182.506760721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls") pod "maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk" (UID: "624705a1-b9fe-4606-96fe-f6d993bcd422") : secret "maas-api-e2e-ait-baf5781e-serving-cert" not found
Oct 30 00:58:21.935838 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.935782    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624705a1-b9fe-4606-96fe-f6d993bcd422-openshift-service-ca\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.946942 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.940308    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-metrics-certs\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:21.958830 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.958736    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:21.960914 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:21.960872    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d622\" (UniqueName: \"kubernetes.io/projected/624705a1-b9fe-4606-96fe-f6d993bcd422-kube-api-access-2d622\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:22.114336 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:22.113858    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:22.441941 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:22.441904    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:22.444572 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:22.444541    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls\") pod \"maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:22.673668 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:22.673628    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:22.992059 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:22.992026    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"]
Oct 30 00:58:23.491224 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:23.491183    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk" event={"ID":"624705a1-b9fe-4606-96fe-f6d993bcd422","Type":"ContainerStarted","Data":"078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377"}
Oct 30 00:58:23.491403 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:23.491261    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk" event={"ID":"624705a1-b9fe-4606-96fe-f6d993bcd422","Type":"ContainerStarted","Data":"5e0769e3e56677edf133be1f834a4ee35472149fdcbdb16a4d2c91e7c543098a"}
Oct 30 00:58:23.491403 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:23.491288    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:23.534714 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:23.534659    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk" podStartSLOduration=2.220657448 podStartE2EDuration="2.534644461s" totalImagesPullingTime="313.987013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:23.533272504 +0000 UTC m=+1183.604596339" watchObservedRunningTime="2026-10-30 00:58:23.534644461 +0000 UTC m=+1183.605968295"
Oct 30 00:58:23.815875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:23.815846    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:25.456334 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:25.456299    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:26.322658 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:26.322615    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:26.458157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:26.458112    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:27.186863 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:27.186819    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:27.349741 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:27.349697    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:27.490051 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:27.489977    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:28.669478 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:28.669430    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:29.154782 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:29.154748    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:29.503313 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:29.503284    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:29.593503 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:29.593449    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:29.740259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:29.739915    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:30.414739 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:30.414703    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:31.161590 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:31.161560    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:32.210720 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:32.210684    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:32.318080 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:32.318034    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:33.212623 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:33.212538    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:33.329510 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:33.329470    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:33.539204 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:33.539116    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:33.654956 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:33.654921    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:33.988616 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:33.988581    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:34.877476 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:34.877435    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:35.010687 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:35.010656    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:36.215624 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:36.215574    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:38.158167 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:38.158133    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:40.492258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:40.491754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:42.026247 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:42.026209    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"]
Oct 30 00:58:42.040897 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:42.040867    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00335efb-g2phd"]
Oct 30 00:58:42.109616 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:42.109582    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd092b5-f93a-4f2f-9482-c0c4e242fa7b" path="/var/lib/kubelet/pods/efd092b5-f93a-4f2f-9482-c0c4e242fa7b/volumes"
Oct 30 00:58:42.641192 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:42.641155    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:44.542128 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:44.542081    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:46.122944 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:46.122910    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:47.831360 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:47.831326    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:48.382366 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:58:48.382329    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:58:49.573802 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:49.573758    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:49.772684 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:49.772649    4576 generic.go:356] "Generic (PLEG): container finished" podID="bcd54faa-802b-4f59-b38d-58921167451a" containerID="44b2a496224a459b97731c1cf7a2d9c5b550c3af905ed8b3d8dc3c55fd49263f" exitCode=0
Oct 30 00:58:49.772893 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:49.772718    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" event={"ID":"bcd54faa-802b-4f59-b38d-58921167451a","Type":"ContainerDied","Data":"44b2a496224a459b97731c1cf7a2d9c5b550c3af905ed8b3d8dc3c55fd49263f"}
Oct 30 00:58:49.773275 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:49.773222    4576 scope.go:122] "RemoveContainer" containerID="44b2a496224a459b97731c1cf7a2d9c5b550c3af905ed8b3d8dc3c55fd49263f"
Oct 30 00:58:49.774865 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:49.774838    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:58:49.774935 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:49.774909    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:58:49.774978 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:49.774940    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:58:50.801785 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:50.801753    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" event={"ID":"bcd54faa-802b-4f59-b38d-58921167451a","Type":"ContainerStarted","Data":"c05f7127051edfd053f9f087e83ec600819d2d02fcf355eb6f37591db3b3c6be"}
Oct 30 00:58:51.232948 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:51.232912    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:51.351683 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:51.351596    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:51.459687 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:51.459658    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:51.488319 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:51.480216    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:58:51.488319 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:51.480400    4576 patch_prober.go:29] interesting pod/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:51.488319 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:51.480445    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:52.292727 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:52.292689    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:52.436221 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:52.436184    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:52.479273 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:52.476650    4576 patch_prober.go:29] interesting pod/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:52.479273 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:52.476716    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:53.160833 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:53.160786    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:53.479049 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:53.479009    4576 patch_prober.go:29] interesting pod/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:53.479698 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:53.479668    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:54.280258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:54.279337    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:58:54.488259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:54.485703    4576 patch_prober.go:29] interesting pod/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:54.488259 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:54.485779    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:54.823280 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:54.823245    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:55.491324 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:55.491288    4576 patch_prober.go:29] interesting pod/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:55.491920 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:55.491884    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:56.467824 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:56.467786    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:56.479955 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:56.479930    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:58:56.481102 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:56.481083    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:58:57.785706 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:57.785668    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"]
Oct 30 00:58:57.786213 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:57.785912    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh" podUID="baf488d7-8c8b-4902-bb69-60f1be863f13" containerName="maas-api" containerID="cri-o://bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66" gracePeriod=30
Oct 30 00:58:57.880088 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:57.880007    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:57.985672 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:57.984799    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:58.100305 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.097010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:58.212971 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.212940    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:58.235952 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.235919    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"]
Oct 30 00:58:58.236207 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.236165    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk" podUID="624705a1-b9fe-4606-96fe-f6d993bcd422" containerName="maas-api" containerID="cri-o://078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377" gracePeriod=30
Oct 30 00:58:58.321892 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.321857    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:58.384584 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.384537    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-metrics-certs\") pod \"baf488d7-8c8b-4902-bb69-60f1be863f13\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") "
Oct 30 00:58:58.384762 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.384632    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf488d7-8c8b-4902-bb69-60f1be863f13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf488d7-8c8b-4902-bb69-60f1be863f13-openshift-service-ca\") pod \"baf488d7-8c8b-4902-bb69-60f1be863f13\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") "
Oct 30 00:58:58.384762 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.384725    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf488d7-8c8b-4902-bb69-60f1be863f13-kube-api-access-d8l4f\" (UniqueName: \"kubernetes.io/projected/baf488d7-8c8b-4902-bb69-60f1be863f13-kube-api-access-d8l4f\") pod \"baf488d7-8c8b-4902-bb69-60f1be863f13\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") "
Oct 30 00:58:58.384882 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.384841    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-maas-api-tls\") pod \"baf488d7-8c8b-4902-bb69-60f1be863f13\" (UID: \"baf488d7-8c8b-4902-bb69-60f1be863f13\") "
Oct 30 00:58:58.385426 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.385391    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf488d7-8c8b-4902-bb69-60f1be863f13-openshift-service-ca" pod "baf488d7-8c8b-4902-bb69-60f1be863f13" (UID: "baf488d7-8c8b-4902-bb69-60f1be863f13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:58.387284 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.387254    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-maas-api-tls" pod "baf488d7-8c8b-4902-bb69-60f1be863f13" (UID: "baf488d7-8c8b-4902-bb69-60f1be863f13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:58.387753 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.387712    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-metrics-certs" pod "baf488d7-8c8b-4902-bb69-60f1be863f13" (UID: "baf488d7-8c8b-4902-bb69-60f1be863f13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:58.388002 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.387979    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf488d7-8c8b-4902-bb69-60f1be863f13-kube-api-access-d8l4f" pod "baf488d7-8c8b-4902-bb69-60f1be863f13" (UID: "baf488d7-8c8b-4902-bb69-60f1be863f13"). InnerVolumeSpecName "kube-api-access-d8l4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:58.485944 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.485877    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:58.487278 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.486460    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-maas-api-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:58:58.487278 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.486478    4576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baf488d7-8c8b-4902-bb69-60f1be863f13-metrics-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:58:58.487278 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.486491    4576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf488d7-8c8b-4902-bb69-60f1be863f13-openshift-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:58:58.487278 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.486504    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8l4f\" (UniqueName: \"kubernetes.io/projected/baf488d7-8c8b-4902-bb69-60f1be863f13-kube-api-access-d8l4f\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:58:58.545121 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.544795    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:58.603252 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.603196    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:58.690839 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.690799    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624705a1-b9fe-4606-96fe-f6d993bcd422-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624705a1-b9fe-4606-96fe-f6d993bcd422-openshift-service-ca\") pod \"624705a1-b9fe-4606-96fe-f6d993bcd422\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") "
Oct 30 00:58:58.691024 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.690891    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624705a1-b9fe-4606-96fe-f6d993bcd422-kube-api-access-2d622\" (UniqueName: \"kubernetes.io/projected/624705a1-b9fe-4606-96fe-f6d993bcd422-kube-api-access-2d622\") pod \"624705a1-b9fe-4606-96fe-f6d993bcd422\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") "
Oct 30 00:58:58.691110 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.691071    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-metrics-certs\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-metrics-certs\") pod \"624705a1-b9fe-4606-96fe-f6d993bcd422\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") "
Oct 30 00:58:58.691155 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.691116    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls\") pod \"624705a1-b9fe-4606-96fe-f6d993bcd422\" (UID: \"624705a1-b9fe-4606-96fe-f6d993bcd422\") "
Oct 30 00:58:58.691391 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.691362    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624705a1-b9fe-4606-96fe-f6d993bcd422-openshift-service-ca" pod "624705a1-b9fe-4606-96fe-f6d993bcd422" (UID: "624705a1-b9fe-4606-96fe-f6d993bcd422"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:58.692000 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.691944    4576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624705a1-b9fe-4606-96fe-f6d993bcd422-openshift-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:58:58.693701 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.693636    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624705a1-b9fe-4606-96fe-f6d993bcd422-kube-api-access-2d622" pod "624705a1-b9fe-4606-96fe-f6d993bcd422" (UID: "624705a1-b9fe-4606-96fe-f6d993bcd422"). InnerVolumeSpecName "kube-api-access-2d622". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:58.694348 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.694321    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls" pod "624705a1-b9fe-4606-96fe-f6d993bcd422" (UID: "624705a1-b9fe-4606-96fe-f6d993bcd422"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:58.694424 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.694339    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-metrics-certs" pod "624705a1-b9fe-4606-96fe-f6d993bcd422" (UID: "624705a1-b9fe-4606-96fe-f6d993bcd422"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:58.783005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.781912    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"]
Oct 30 00:58:58.783005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.782104    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy" containerID="cri-o://c05f7127051edfd053f9f087e83ec600819d2d02fcf355eb6f37591db3b3c6be" gracePeriod=30
Oct 30 00:58:58.795168 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.793098    4576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-metrics-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:58:58.795168 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.793120    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624705a1-b9fe-4606-96fe-f6d993bcd422-maas-api-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:58:58.795168 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.793132    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d622\" (UniqueName: \"kubernetes.io/projected/624705a1-b9fe-4606-96fe-f6d993bcd422-kube-api-access-2d622\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:58:58.833552 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.832797    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:58.856626 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.856592    4576 generic.go:356] "Generic (PLEG): container finished" podID="baf488d7-8c8b-4902-bb69-60f1be863f13" containerID="bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66" exitCode=0
Oct 30 00:58:58.856825 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.856667    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh" event={"ID":"baf488d7-8c8b-4902-bb69-60f1be863f13","Type":"ContainerDied","Data":"bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66"}
Oct 30 00:58:58.856825 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.856708    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh" event={"ID":"baf488d7-8c8b-4902-bb69-60f1be863f13","Type":"ContainerDied","Data":"7daeb18f7a47d21cf2685d5d1117b6d897ea66a97be66166953e12604028e003"}
Oct 30 00:58:58.856825 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.856729    4576 scope.go:122] "RemoveContainer" containerID="bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66"
Oct 30 00:58:58.856993 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.856838    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"
Oct 30 00:58:58.866948 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.866106    4576 generic.go:356] "Generic (PLEG): container finished" podID="624705a1-b9fe-4606-96fe-f6d993bcd422" containerID="078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377" exitCode=0
Oct 30 00:58:58.866948 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.866304    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk" event={"ID":"624705a1-b9fe-4606-96fe-f6d993bcd422","Type":"ContainerDied","Data":"078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377"}
Oct 30 00:58:58.866948 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.866354    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk" event={"ID":"624705a1-b9fe-4606-96fe-f6d993bcd422","Type":"ContainerDied","Data":"5e0769e3e56677edf133be1f834a4ee35472149fdcbdb16a4d2c91e7c543098a"}
Oct 30 00:58:58.866948 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.866426    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"
Oct 30 00:58:58.874592 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.874572    4576 scope.go:122] "RemoveContainer" containerID="bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66"
Oct 30 00:58:58.874978 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:58:58.874948    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66\": container with ID starting with bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66 not found: ID does not exist" containerID="bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66"
Oct 30 00:58:58.875076 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.874983    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66"} err="failed to get container status \"bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66\": rpc error: code = NotFound desc = could not find container \"bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66\": container with ID starting with bbe8b660e42a976128f7c7fecd8e4a5b08e8c2046bab03f7941b09e1eafbdc66 not found: ID does not exist"
Oct 30 00:58:58.875076 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.875007    4576 scope.go:122] "RemoveContainer" containerID="078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377"
Oct 30 00:58:58.887490 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.887466    4576 scope.go:122] "RemoveContainer" containerID="078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377"
Oct 30 00:58:58.887823 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:58:58.887802    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377\": container with ID starting with 078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377 not found: ID does not exist" containerID="078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377"
Oct 30 00:58:58.887904 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.887834    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377"} err="failed to get container status \"078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377\": rpc error: code = NotFound desc = could not find container \"078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377\": container with ID starting with 078e86098ca07c50675e91f2e5d7b0a180f82812cb51f469a2da00aa8121f377 not found: ID does not exist"
Oct 30 00:58:58.920523 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.920494    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"]
Oct 30 00:58:58.931761 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:58.931734    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a03b2130-79ffff68fc-w5rnh"]
Oct 30 00:58:59.003359 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:59.003275    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"]
Oct 30 00:58:59.006601 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:59.006561    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-baf5781e-5f8d55b8cc-4nmmk"]
Oct 30 00:58:59.753673 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:59.753640    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:59.870460 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:59.870408    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:58:59.984652 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:58:59.984619    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:00.122831 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:00.122743    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624705a1-b9fe-4606-96fe-f6d993bcd422" path="/var/lib/kubelet/pods/624705a1-b9fe-4606-96fe-f6d993bcd422/volumes"
Oct 30 00:59:00.123371 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:00.123350    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf488d7-8c8b-4902-bb69-60f1be863f13" path="/var/lib/kubelet/pods/baf488d7-8c8b-4902-bb69-60f1be863f13/volumes"
Oct 30 00:59:00.285152 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:00.285118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:01.966994 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:01.966962    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:03.458746 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:03.458712    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:03.901057 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:03.901024    4576 generic.go:356] "Generic (PLEG): container finished" podID="bcd54faa-802b-4f59-b38d-58921167451a" containerID="c05f7127051edfd053f9f087e83ec600819d2d02fcf355eb6f37591db3b3c6be" exitCode=0
Oct 30 00:59:03.901213 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:03.901096    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" event={"ID":"bcd54faa-802b-4f59-b38d-58921167451a","Type":"ContainerDied","Data":"c05f7127051edfd053f9f087e83ec600819d2d02fcf355eb6f37591db3b3c6be"}
Oct 30 00:59:03.901213 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:03.901155    4576 scope.go:122] "RemoveContainer" containerID="44b2a496224a459b97731c1cf7a2d9c5b550c3af905ed8b3d8dc3c55fd49263f"
Oct 30 00:59:04.078058 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.078034    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:59:04.166018 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.165990    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-data\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166205 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166040    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-certs\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166205 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166110    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-socket\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166205 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166154    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-credential-socket\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166205 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166189    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcd54faa-802b-4f59-b38d-58921167451a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd54faa-802b-4f59-b38d-58921167451a-istio-podinfo\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166329    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-kube-api-access-d9tm8\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-kube-api-access-d9tm8\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166376    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-data" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:04.166452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166389    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-certs" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:04.166452 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166384    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-istio-token\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-istio-token\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166467    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-socket" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:04.166665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166511    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd54faa-802b-4f59-b38d-58921167451a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd54faa-802b-4f59-b38d-58921167451a-istiod-ca-cert\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166665 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166586    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-envoy\") pod \"bcd54faa-802b-4f59-b38d-58921167451a\" (UID: \"bcd54faa-802b-4f59-b38d-58921167451a\") "
Oct 30 00:59:04.166829 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166671    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-credential-socket" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:04.166988 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.166960    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd54faa-802b-4f59-b38d-58921167451a-istiod-ca-cert" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:04.168579 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.168550    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd54faa-802b-4f59-b38d-58921167451a-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.168579 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.168581    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.168915 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.168594    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.168915 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.168609    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.168915 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.168621    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.169530 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.169505    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-kube-api-access-d9tm8" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "kube-api-access-d9tm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:04.169636 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.169534    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-istio-token" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:04.170147 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.170113    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcd54faa-802b-4f59-b38d-58921167451a-istio-podinfo" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:04.170517 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.170499    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-envoy" pod "bcd54faa-802b-4f59-b38d-58921167451a" (UID: "bcd54faa-802b-4f59-b38d-58921167451a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:04.269875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.269846    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9tm8\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-kube-api-access-d9tm8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.269875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.269872    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd54faa-802b-4f59-b38d-58921167451a-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.269875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.269882    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd54faa-802b-4f59-b38d-58921167451a-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.270111 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.269890    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd54faa-802b-4f59-b38d-58921167451a-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.298620 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.298584    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:59:04.908525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.908494    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"
Oct 30 00:59:04.908525 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.908515    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz" event={"ID":"bcd54faa-802b-4f59-b38d-58921167451a","Type":"ContainerDied","Data":"675f3502f74ac81c70caeb98f6dcfdeecf581fc951df24134958139251da7daf"}
Oct 30 00:59:04.909095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.908614    4576 scope.go:122] "RemoveContainer" containerID="c05f7127051edfd053f9f087e83ec600819d2d02fcf355eb6f37591db3b3c6be"
Oct 30 00:59:04.981028 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.980998    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"]
Oct 30 00:59:04.994633 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:04.994602    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a03b2130-openshift-default-59b655b4f-4lrjz"]
Oct 30 00:59:05.059551 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:05.059520    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:06.108661 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:06.108622    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd54faa-802b-4f59-b38d-58921167451a" path="/var/lib/kubelet/pods/bcd54faa-802b-4f59-b38d-58921167451a/volumes"
Oct 30 00:59:06.949777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:06.949742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:08.780610 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:08.780575    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:10.912768 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:10.912729    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:11.864570 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:11.864533    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:12.703028 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:12.702996    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:14.323090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:14.323047    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:14.499359 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:14.499221    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:14.608082 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:14.607994    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:16.374994 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:16.374959    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:16.488469 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:16.488424    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:16.594136 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:16.594092    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:16.707222 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:16.707185    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:17.290808 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.290771    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:17.416537 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.416500    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:17.611876 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.611789    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:17.965482 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.965445    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"]
Oct 30 00:59:17.966582 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.966554    4576 cpu_manager.go:414] "containerMap: removing container" podUID="624705a1-b9fe-4606-96fe-f6d993bcd422" containerName="maas-api"
Oct 30 00:59:17.966767 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.966743    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624705a1-b9fe-4606-96fe-f6d993bcd422" containerName="maas-api"
Oct 30 00:59:17.966902 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.966796    4576 cpu_manager.go:414] "containerMap: removing container" podUID="baf488d7-8c8b-4902-bb69-60f1be863f13" containerName="maas-api"
Oct 30 00:59:17.966902 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.966805    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf488d7-8c8b-4902-bb69-60f1be863f13" containerName="maas-api"
Oct 30 00:59:17.966902 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.966824    4576 cpu_manager.go:414] "containerMap: removing container" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy"
Oct 30 00:59:17.966902 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.966830    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy"
Oct 30 00:59:17.967114 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.966985    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="624705a1-b9fe-4606-96fe-f6d993bcd422" containerName="maas-api"
Oct 30 00:59:17.967114 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.966999    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf488d7-8c8b-4902-bb69-60f1be863f13" containerName="maas-api"
Oct 30 00:59:17.967114 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.967008    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy"
Oct 30 00:59:17.967114 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:17.967017    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy"
Oct 30 00:59:18.008875 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.008843    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"]
Oct 30 00:59:18.011502 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.011455    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.038765 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.038743    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-95b31f79-gw-openshift-default-dockercfg-qd47z\""
Oct 30 00:59:18.050935 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.050901    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:18.173132 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.173087    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-data\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.174361 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.174333    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-token\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.174643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.174627    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-podinfo\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.178262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.174759    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p485z\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-kube-api-access-p485z\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.179409 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.179383    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-certs\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.179678 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.179660    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-envoy\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.179813 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.179798    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cb575-2086-46cc-97c4-648c170bf98e-istiod-ca-cert\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.179924 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.179908    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-credential-socket\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.180077 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.180062    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-socket\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.273705 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.273612    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:18.281546 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281484    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-podinfo\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.281743 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281584    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p485z\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-kube-api-access-p485z\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.281743 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281634    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-certs\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.281861 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281753    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-envoy\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.281861 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281776    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cb575-2086-46cc-97c4-648c170bf98e-istiod-ca-cert\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.281861 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281799    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-credential-socket\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.282012 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281861    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-socket\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.282012 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281911    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-data\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.282012 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.281928    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-token\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.282928 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.282901    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-data\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.283094 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.283068    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cb575-2086-46cc-97c4-648c170bf98e-istiod-ca-cert\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.283181 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.283089    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-credential-socket\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.283181 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.282962    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-certs\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.283415 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.283386    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-socket\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.285209 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.285185    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-envoy\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.286691 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.286666    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-podinfo\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.303588 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.303563    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-token\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.304090 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.304072    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p485z\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-kube-api-access-p485z\") pod \"e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") " pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.341004 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.340975    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:18.399046 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:59:18.399011    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:59:18.652981 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.652884    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:18.672762 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:59:18.672728    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3cb575_2086_46cc_97c4_648c170bf98e.slice/crio-2e1939f07dbfb55ef606b3735eeea69cc71a3c5543c36bcc1b0d8f08239a3c66 WatchSource:0}: Error finding container 2e1939f07dbfb55ef606b3735eeea69cc71a3c5543c36bcc1b0d8f08239a3c66: Status 404 returned error can't find the container with id 2e1939f07dbfb55ef606b3735eeea69cc71a3c5543c36bcc1b0d8f08239a3c66
Oct 30 00:59:18.672936 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.672869    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"]
Oct 30 00:59:18.675933 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.675896    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:59:18.676036 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.675967    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:59:18.676036 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.675996    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 00:59:18.769262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.769207    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:18.902859 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:18.902822    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:19.011671 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:19.011627    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72" event={"ID":"ff3cb575-2086-46cc-97c4-648c170bf98e","Type":"ContainerStarted","Data":"539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186"}
Oct 30 00:59:19.011884 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:19.011692    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72" event={"ID":"ff3cb575-2086-46cc-97c4-648c170bf98e","Type":"ContainerStarted","Data":"2e1939f07dbfb55ef606b3735eeea69cc71a3c5543c36bcc1b0d8f08239a3c66"}
Oct 30 00:59:19.033776 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:19.033738    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:19.057593 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:19.057534    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72" podStartSLOduration=2.057519542 podStartE2EDuration="2.057519542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:19.055525463 +0000 UTC m=+1239.126849299" watchObservedRunningTime="2026-10-30 00:59:19.057519542 +0000 UTC m=+1239.128843377"
Oct 30 00:59:19.143885 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:19.143852    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:19.341374 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:19.341283    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:19.347603 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:19.347577    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:20.018723 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:20.018692    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:20.019777 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:20.019758    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 00:59:20.872696 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:20.872662    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:20.980568 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:20.980525    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:21.765651 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.764373    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:21.890871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.890835    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:21.926482 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.926448    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"]
Oct 30 00:59:21.927057 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.927030    4576 cpu_manager.go:414] "containerMap: removing container" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy"
Oct 30 00:59:21.927057 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.927048    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd54faa-802b-4f59-b38d-58921167451a" containerName="istio-proxy"
Oct 30 00:59:21.930781 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.930694    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:21.948112 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.948081    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"]
Oct 30 00:59:21.957755 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.957728    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-95b31f79\""
Oct 30 00:59:21.973652 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:21.973619    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-95b31f79-dockercfg-8gkvw\""
Oct 30 00:59:22.030415 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.018283    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"]
Oct 30 00:59:22.090260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.070656    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjbr\" (UniqueName: \"kubernetes.io/projected/04064695-2bba-44c9-a11c-82ba8afd80fb-kube-api-access-dpjbr\") pod \"payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs\" (UID: \"04064695-2bba-44c9-a11c-82ba8afd80fb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:22.090260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.070840    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04064695-2bba-44c9-a11c-82ba8afd80fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs\" (UID: \"04064695-2bba-44c9-a11c-82ba8afd80fb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:22.158933 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.153512    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"]
Oct 30 00:59:22.158933 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.157824    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:22.173717 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.173673    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjbr\" (UniqueName: \"kubernetes.io/projected/04064695-2bba-44c9-a11c-82ba8afd80fb-kube-api-access-dpjbr\") pod \"payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs\" (UID: \"04064695-2bba-44c9-a11c-82ba8afd80fb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:22.173869 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.173839    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04064695-2bba-44c9-a11c-82ba8afd80fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs\" (UID: \"04064695-2bba-44c9-a11c-82ba8afd80fb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:22.176168 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.176137    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04064695-2bba-44c9-a11c-82ba8afd80fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs\" (UID: \"04064695-2bba-44c9-a11c-82ba8afd80fb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:22.195221 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.195184    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjbr\" (UniqueName: \"kubernetes.io/projected/04064695-2bba-44c9-a11c-82ba8afd80fb-kube-api-access-dpjbr\") pod \"payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs\" (UID: \"04064695-2bba-44c9-a11c-82ba8afd80fb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:22.246413 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.246382    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:22.277095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.277053    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-plugins-config-volume\") pod \"payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md\" (UID: \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\") " pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:22.278915 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.277638    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntj4c\" (UniqueName: \"kubernetes.io/projected/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-kube-api-access-ntj4c\") pod \"payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md\" (UID: \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\") " pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:22.379534 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.379498    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-plugins-config-volume\") pod \"payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md\" (UID: \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\") " pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:22.379752 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.379664    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntj4c\" (UniqueName: \"kubernetes.io/projected/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-kube-api-access-ntj4c\") pod \"payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md\" (UID: \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\") " pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:22.380399 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.380378    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-plugins-config-volume\") pod \"payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md\" (UID: \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\") " pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:22.402817 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.402783    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntj4c\" (UniqueName: \"kubernetes.io/projected/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-kube-api-access-ntj4c\") pod \"payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md\" (UID: \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\") " pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:22.485497 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.485458    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:22.562321 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.561185    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"]
Oct 30 00:59:22.566909 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:59:22.566879    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04064695_2bba_44c9_a11c_82ba8afd80fb.slice/crio-a5cf33193abbac0b7eacc407c3bb0a2fda867ae939ca14c2cf146c35fc219564 WatchSource:0}: Error finding container a5cf33193abbac0b7eacc407c3bb0a2fda867ae939ca14c2cf146c35fc219564: Status 404 returned error can't find the container with id a5cf33193abbac0b7eacc407c3bb0a2fda867ae939ca14c2cf146c35fc219564
Oct 30 00:59:22.786993 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.786967    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"]
Oct 30 00:59:22.788159 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:59:22.788134    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode071f5bc_95c2_4de2_b4d7_23aae2fb53df.slice/crio-16c1c0edb792f87d1a4966c938d172d23966b4aea3f8a1ffbb5cc6cdf933a05a WatchSource:0}: Error finding container 16c1c0edb792f87d1a4966c938d172d23966b4aea3f8a1ffbb5cc6cdf933a05a: Status 404 returned error can't find the container with id 16c1c0edb792f87d1a4966c938d172d23966b4aea3f8a1ffbb5cc6cdf933a05a
Oct 30 00:59:22.888211 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:22.888179    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:23.188882 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:23.188847    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md" event={"ID":"e071f5bc-95c2-4de2-b4d7-23aae2fb53df","Type":"ContainerStarted","Data":"613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113"}
Oct 30 00:59:23.189095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:23.188904    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md" event={"ID":"e071f5bc-95c2-4de2-b4d7-23aae2fb53df","Type":"ContainerStarted","Data":"16c1c0edb792f87d1a4966c938d172d23966b4aea3f8a1ffbb5cc6cdf933a05a"}
Oct 30 00:59:23.189095 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:23.188932    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:23.190645 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:23.190616    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs" event={"ID":"04064695-2bba-44c9-a11c-82ba8afd80fb","Type":"ContainerStarted","Data":"bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce"}
Oct 30 00:59:23.190788 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:23.190664    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs" event={"ID":"04064695-2bba-44c9-a11c-82ba8afd80fb","Type":"ContainerStarted","Data":"a5cf33193abbac0b7eacc407c3bb0a2fda867ae939ca14c2cf146c35fc219564"}
Oct 30 00:59:23.190788 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:23.190686    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:23.236556 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:23.236508    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md" podStartSLOduration=2.23649205 podStartE2EDuration="2.23649205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:23.233281818 +0000 UTC m=+1243.304605654" watchObservedRunningTime="2026-10-30 00:59:23.23649205 +0000 UTC m=+1243.307815884"
Oct 30 00:59:23.278215 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:23.278157    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs" podStartSLOduration=2.278142286 podStartE2EDuration="2.278142286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:23.274795679 +0000 UTC m=+1243.346119511" watchObservedRunningTime="2026-10-30 00:59:23.278142286 +0000 UTC m=+1243.349466122"
Oct 30 00:59:24.690175 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:24.690134    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:24.810326 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:24.810289    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:25.386070 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:25.386021    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:25.522366 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:25.522336    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:26.011126 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:26.011085    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:26.142908 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:26.142869    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:26.427433 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:26.427340    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:26.926341 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:26.926302    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:27.027832 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:27.027800    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:28.693588 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:28.693544    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:30.097817 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:30.097774    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:30.231171 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:30.231141    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:30.369954 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:30.369868    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:30.503290 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:30.503257    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 00:59:31.291258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:31.290056    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:32.029438 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:32.029400    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:32.152031 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:32.151998    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:32.320935 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:32.320853    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:33.673020 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:33.672985    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:33.774933 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:33.774899    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:34.204323 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:34.204295    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 00:59:34.206701 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:34.206680    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 00:59:34.371079 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:34.371047    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:35.433634 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:35.433603    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:35.545391 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:35.545360    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:35.649581 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:35.649542    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:36.972294 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:36.972252    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:38.580657 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:38.580630    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:40.189552 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:40.189515    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:40.233888 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:40.233508    4576 scope.go:122] "RemoveContainer" containerID="851b7a555c05197d83b56d258a583915a28da706e5e245a5085f4b7c63fdc774"
Oct 30 00:59:41.041303 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:41.041267    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:41.154937 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:41.154902    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:41.921406 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:41.921376    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:42.048248 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:42.048195    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:42.159860 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:42.159814    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:43.583809 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:43.583765    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:44.144386 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:44.144351    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:44.445157 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:44.445122    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:44.568103 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:44.568063    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:44.682258 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:44.682188    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:45.628005 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:45.627951    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:45.759016 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:45.758951    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:45.812716 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:45.812683    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"]
Oct 30 00:59:45.818224 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:45.818204    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:45.837646 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:45.837619    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"]
Oct 30 00:59:45.845776 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:45.845752    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-ca324d-serving-cert\""
Oct 30 00:59:45.883643 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:45.883559    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:46.012081 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.012045    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:46.012328 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.012092    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w94qs\" (UniqueName: \"kubernetes.io/projected/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-kube-api-access-w94qs\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.012402 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.012343    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.012472 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.012441    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.012540 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.012523    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.114637 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.114609    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w94qs\" (UniqueName: \"kubernetes.io/projected/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-kube-api-access-w94qs\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.114808 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.114723    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.114851 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.114816    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.114908 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.114891    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.115763 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.115698    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.117522 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.117497    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.117640 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.117537    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.131147 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.131117    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:46.135909 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.135845    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w94qs\" (UniqueName: \"kubernetes.io/projected/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-kube-api-access-w94qs\") pod \"maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.433357 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.433259    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:46.733068 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:46.733043    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"]
Oct 30 00:59:46.734852 ip-10-0-130-5 kubenswrapper[4576]: W1030 00:59:46.734824    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d892ed_cfd0_4c31_b52e_c11f800cd7ce.slice/crio-f766683e72e0c187b23bdadb791cbdd627a5ca18fe4ca6022decae30b3d545f0 WatchSource:0}: Error finding container f766683e72e0c187b23bdadb791cbdd627a5ca18fe4ca6022decae30b3d545f0: Status 404 returned error can't find the container with id f766683e72e0c187b23bdadb791cbdd627a5ca18fe4ca6022decae30b3d545f0
Oct 30 00:59:47.425323 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:47.425215    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8" event={"ID":"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce","Type":"ContainerStarted","Data":"9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84"}
Oct 30 00:59:47.425323 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:47.425291    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8" event={"ID":"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce","Type":"ContainerStarted","Data":"f766683e72e0c187b23bdadb791cbdd627a5ca18fe4ca6022decae30b3d545f0"}
Oct 30 00:59:47.425731 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:47.425437    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:47.471108 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:47.471049    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8" podStartSLOduration=2.126084005 podStartE2EDuration="2.471034455s" totalImagesPullingTime="344.95045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:47.468688966 +0000 UTC m=+1267.540012811" watchObservedRunningTime="2026-10-30 00:59:47.471034455 +0000 UTC m=+1267.542358290"
Oct 30 00:59:48.017871 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:48.017833    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:48.381579 ip-10-0-130-5 kubenswrapper[4576]: E1030 00:59:48.381500    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 00:59:49.675955 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:49.675908    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:51.011371 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:51.011334    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:51.158640 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:51.158586    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:52.890053 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:52.890012    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:53.437502 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:53.437469    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 00:59:54.363538 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:54.363500    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:56.221262 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:56.220748    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:56.329257 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:56.326957    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:56.453281 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:56.453197    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:57.291260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:57.290340    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-fjp5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:59:57.291260 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:57.290412    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:58.265200 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:58.265163    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 00:59:59.052983 ip-10-0-130-5 kubenswrapper[4576]: I1030 00:59:59.052943    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:00.148997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.148964    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"]
Oct 30 01:00:00.153022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.152999    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"]
Oct 30 01:00:00.153145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.153112    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:00.157325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.157297    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:00.168908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.168881    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"]
Oct 30 01:00:00.177808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.177777    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"]
Oct 30 01:00:00.177936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.177876    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:00.182434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.182406    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:00:00.182581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.182438    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"]
Oct 30 01:00:00.191907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.191829    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89l2\" (UniqueName: \"kubernetes.io/projected/6d47014e-b95a-4990-8ef0-eabb2dd738a1-kube-api-access-v89l2\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd\" (UID: \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:00.192042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.191903    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47014e-b95a-4990-8ef0-eabb2dd738a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd\" (UID: \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:00.192042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.191989    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xw6\" (UniqueName: \"kubernetes.io/projected/24b711ee-52f3-4747-b43c-4e68af0a6250-kube-api-access-59xw6\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49\" (UID: \"24b711ee-52f3-4747-b43c-4e68af0a6250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:00.192207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.192100    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b711ee-52f3-4747-b43c-4e68af0a6250-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49\" (UID: \"24b711ee-52f3-4747-b43c-4e68af0a6250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:00.192207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.192140    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b987bdd2-bc91-4ac0-81b7-814a273ecb52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5\" (UID: \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:00.192681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.192639    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx95w\" (UniqueName: \"kubernetes.io/projected/b987bdd2-bc91-4ac0-81b7-814a273ecb52-kube-api-access-dx95w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5\" (UID: \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:00.211436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.211189    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"]
Oct 30 01:00:00.211436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.211287    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"]
Oct 30 01:00:00.211436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.211321    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"]
Oct 30 01:00:00.211713 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.211450    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:00.293717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.293681    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b711ee-52f3-4747-b43c-4e68af0a6250-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49\" (UID: \"24b711ee-52f3-4747-b43c-4e68af0a6250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:00.293906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.293734    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b987bdd2-bc91-4ac0-81b7-814a273ecb52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5\" (UID: \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:00.293906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.293770    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx95w\" (UniqueName: \"kubernetes.io/projected/b987bdd2-bc91-4ac0-81b7-814a273ecb52-kube-api-access-dx95w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5\" (UID: \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:00.294059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.293970    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89l2\" (UniqueName: \"kubernetes.io/projected/6d47014e-b95a-4990-8ef0-eabb2dd738a1-kube-api-access-v89l2\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd\" (UID: \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:00.294059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.294025    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47014e-b95a-4990-8ef0-eabb2dd738a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd\" (UID: \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:00.294059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.294055    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgn9\" (UniqueName: \"kubernetes.io/projected/275f57a2-15c2-484d-8175-b35b6bf2930d-kube-api-access-7vgn9\") pod \"maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82\" (UID: \"275f57a2-15c2-484d-8175-b35b6bf2930d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:00.294220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.294088    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/275f57a2-15c2-484d-8175-b35b6bf2930d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82\" (UID: \"275f57a2-15c2-484d-8175-b35b6bf2930d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:00.294220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.294166    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59xw6\" (UniqueName: \"kubernetes.io/projected/24b711ee-52f3-4747-b43c-4e68af0a6250-kube-api-access-59xw6\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49\" (UID: \"24b711ee-52f3-4747-b43c-4e68af0a6250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:00.294463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.294441    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b711ee-52f3-4747-b43c-4e68af0a6250-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49\" (UID: \"24b711ee-52f3-4747-b43c-4e68af0a6250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:00.294560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.294540    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b987bdd2-bc91-4ac0-81b7-814a273ecb52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5\" (UID: \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:00.294774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.294752    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47014e-b95a-4990-8ef0-eabb2dd738a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd\" (UID: \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:00.314879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.314855    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xw6\" (UniqueName: \"kubernetes.io/projected/24b711ee-52f3-4747-b43c-4e68af0a6250-kube-api-access-59xw6\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49\" (UID: \"24b711ee-52f3-4747-b43c-4e68af0a6250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:00.315027 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.314951    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89l2\" (UniqueName: \"kubernetes.io/projected/6d47014e-b95a-4990-8ef0-eabb2dd738a1-kube-api-access-v89l2\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd\" (UID: \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:00.315099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.315080    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx95w\" (UniqueName: \"kubernetes.io/projected/b987bdd2-bc91-4ac0-81b7-814a273ecb52-kube-api-access-dx95w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5\" (UID: \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:00.395671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.395637    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgn9\" (UniqueName: \"kubernetes.io/projected/275f57a2-15c2-484d-8175-b35b6bf2930d-kube-api-access-7vgn9\") pod \"maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82\" (UID: \"275f57a2-15c2-484d-8175-b35b6bf2930d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:00.395861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.395685    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/275f57a2-15c2-484d-8175-b35b6bf2930d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82\" (UID: \"275f57a2-15c2-484d-8175-b35b6bf2930d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:00.396295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.396277    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/275f57a2-15c2-484d-8175-b35b6bf2930d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82\" (UID: \"275f57a2-15c2-484d-8175-b35b6bf2930d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:00.416530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.416458    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgn9\" (UniqueName: \"kubernetes.io/projected/275f57a2-15c2-484d-8175-b35b6bf2930d-kube-api-access-7vgn9\") pod \"maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82\" (UID: \"275f57a2-15c2-484d-8175-b35b6bf2930d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:00.474493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.474458    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:00.480810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.480787    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:00.507103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.507077    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:00.530076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.530043    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:00.784448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.784423    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"]
Oct 30 01:00:00.796159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.796135    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"]
Oct 30 01:00:00.799883 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:00:00.799848    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d47014e_b95a_4990_8ef0_eabb2dd738a1.slice/crio-a5f43369d24aee70c20136bafe1f6b5c21f5b48fa60a410ea44c172aed467051 WatchSource:0}: Error finding container a5f43369d24aee70c20136bafe1f6b5c21f5b48fa60a410ea44c172aed467051: Status 404 returned error can't find the container with id a5f43369d24aee70c20136bafe1f6b5c21f5b48fa60a410ea44c172aed467051
Oct 30 01:00:00.817580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.817554    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:00.829963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.829940    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"]
Oct 30 01:00:00.830822 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:00:00.830798    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb987bdd2_bc91_4ac0_81b7_814a273ecb52.slice/crio-690c5a7112b14d64272cb81cd2c637331951dfeb2812e8ea7f5518e2f5d893f2 WatchSource:0}: Error finding container 690c5a7112b14d64272cb81cd2c637331951dfeb2812e8ea7f5518e2f5d893f2: Status 404 returned error can't find the container with id 690c5a7112b14d64272cb81cd2c637331951dfeb2812e8ea7f5518e2f5d893f2
Oct 30 01:00:00.866641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:00.866615    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"]
Oct 30 01:00:01.629062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.628967    4576 generic.go:356] "Generic (PLEG): container finished" podID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerID="9f38e6f7c18e471f159ead483f7f8926a43cc7eec49d9631f0be415bc7048c99" exitCode=60
Oct 30 01:00:01.629520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.629061    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" event={"ID":"b987bdd2-bc91-4ac0-81b7-814a273ecb52","Type":"ContainerDied","Data":"9f38e6f7c18e471f159ead483f7f8926a43cc7eec49d9631f0be415bc7048c99"}
Oct 30 01:00:01.629520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.629152    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" event={"ID":"b987bdd2-bc91-4ac0-81b7-814a273ecb52","Type":"ContainerStarted","Data":"690c5a7112b14d64272cb81cd2c637331951dfeb2812e8ea7f5518e2f5d893f2"}
Oct 30 01:00:01.629520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.629352    4576 scope.go:122] "RemoveContainer" containerID="9f38e6f7c18e471f159ead483f7f8926a43cc7eec49d9631f0be415bc7048c99"
Oct 30 01:00:01.630951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.630895    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" event={"ID":"24b711ee-52f3-4747-b43c-4e68af0a6250","Type":"ContainerStarted","Data":"5bd2ade663013a3dcf6ed8ed93949b496997297d46ca0fbf10ec203da79c7d68"}
Oct 30 01:00:01.630951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.630936    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" event={"ID":"24b711ee-52f3-4747-b43c-4e68af0a6250","Type":"ContainerStarted","Data":"021a61144574a65b8157837319931477a0e52932e627b279f30c56fe84fb5a2c"}
Oct 30 01:00:01.632597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.632576    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" event={"ID":"275f57a2-15c2-484d-8175-b35b6bf2930d","Type":"ContainerStarted","Data":"74ebbcf36d84351d9a879800f48f38ea27033c88d0d898d544ef03e8df954c71"}
Oct 30 01:00:01.632671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.632611    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" event={"ID":"275f57a2-15c2-484d-8175-b35b6bf2930d","Type":"ContainerStarted","Data":"e756cd615981d7f0d12208b7afed5f8430e247e8d852a91bb9c14d67fc8ba483"}
Oct 30 01:00:01.634291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.634272    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" event={"ID":"6d47014e-b95a-4990-8ef0-eabb2dd738a1","Type":"ContainerStarted","Data":"7da62b527b6f641d9843dc500339997a7ce8f907b954a6333284d6e682a4c69b"}
Oct 30 01:00:01.634387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.634307    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" event={"ID":"6d47014e-b95a-4990-8ef0-eabb2dd738a1","Type":"ContainerStarted","Data":"a5f43369d24aee70c20136bafe1f6b5c21f5b48fa60a410ea44c172aed467051"}
Oct 30 01:00:01.712594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.712540    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" podStartSLOduration=1.712525405 podStartE2EDuration="1.712525405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.70959639 +0000 UTC m=+1281.780920238" watchObservedRunningTime="2026-10-30 01:00:01.712525405 +0000 UTC m=+1281.783849239"
Oct 30 01:00:01.751661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.751602    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" podStartSLOduration=1.751591592 podStartE2EDuration="1.751591592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.750457681 +0000 UTC m=+1281.821781509" watchObservedRunningTime="2026-10-30 01:00:01.751591592 +0000 UTC m=+1281.822915488"
Oct 30 01:00:01.795095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:01.795048    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" podStartSLOduration=1.79503484 podStartE2EDuration="1.79503484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:01.791974018 +0000 UTC m=+1281.863297850" watchObservedRunningTime="2026-10-30 01:00:01.79503484 +0000 UTC m=+1281.866358675"
Oct 30 01:00:02.458965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.458928    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:02.642484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.642392    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerID="7da62b527b6f641d9843dc500339997a7ce8f907b954a6333284d6e682a4c69b" exitCode=60
Oct 30 01:00:02.642484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.642472    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" event={"ID":"6d47014e-b95a-4990-8ef0-eabb2dd738a1","Type":"ContainerDied","Data":"7da62b527b6f641d9843dc500339997a7ce8f907b954a6333284d6e682a4c69b"}
Oct 30 01:00:02.642977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.642759    4576 scope.go:122] "RemoveContainer" containerID="7da62b527b6f641d9843dc500339997a7ce8f907b954a6333284d6e682a4c69b"
Oct 30 01:00:02.644997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.644976    4576 generic.go:356] "Generic (PLEG): container finished" podID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerID="fe236289769223f7aa067692a09f4e968f053cd79079dd307350d89236c06cff" exitCode=60
Oct 30 01:00:02.645099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.645046    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" event={"ID":"b987bdd2-bc91-4ac0-81b7-814a273ecb52","Type":"ContainerDied","Data":"fe236289769223f7aa067692a09f4e968f053cd79079dd307350d89236c06cff"}
Oct 30 01:00:02.645160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.645140    4576 scope.go:122] "RemoveContainer" containerID="9f38e6f7c18e471f159ead483f7f8926a43cc7eec49d9631f0be415bc7048c99"
Oct 30 01:00:02.645471 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.645454    4576 scope.go:122] "RemoveContainer" containerID="fe236289769223f7aa067692a09f4e968f053cd79079dd307350d89236c06cff"
Oct 30 01:00:02.645990 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:02.645967    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5_odh-ai-gateway-infra(b987bdd2-bc91-4ac0-81b7-814a273ecb52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52"
Oct 30 01:00:02.650808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.650774    4576 generic.go:356] "Generic (PLEG): container finished" podID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerID="5bd2ade663013a3dcf6ed8ed93949b496997297d46ca0fbf10ec203da79c7d68" exitCode=60
Oct 30 01:00:02.650907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.650867    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" event={"ID":"24b711ee-52f3-4747-b43c-4e68af0a6250","Type":"ContainerDied","Data":"5bd2ade663013a3dcf6ed8ed93949b496997297d46ca0fbf10ec203da79c7d68"}
Oct 30 01:00:02.651160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.651144    4576 scope.go:122] "RemoveContainer" containerID="5bd2ade663013a3dcf6ed8ed93949b496997297d46ca0fbf10ec203da79c7d68"
Oct 30 01:00:02.653607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.653581    4576 generic.go:356] "Generic (PLEG): container finished" podID="275f57a2-15c2-484d-8175-b35b6bf2930d" containerID="74ebbcf36d84351d9a879800f48f38ea27033c88d0d898d544ef03e8df954c71" exitCode=60
Oct 30 01:00:02.653702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.653633    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" event={"ID":"275f57a2-15c2-484d-8175-b35b6bf2930d","Type":"ContainerDied","Data":"74ebbcf36d84351d9a879800f48f38ea27033c88d0d898d544ef03e8df954c71"}
Oct 30 01:00:02.654081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.654055    4576 scope.go:122] "RemoveContainer" containerID="74ebbcf36d84351d9a879800f48f38ea27033c88d0d898d544ef03e8df954c71"
Oct 30 01:00:02.712521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.712491    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:02.803885 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.803848    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"]
Oct 30 01:00:02.804086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:02.804065    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy" containerID="cri-o://d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992" gracePeriod=30
Oct 30 01:00:03.216348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:03.216311    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:03.420648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:03.420525    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:03.664289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:03.664223    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" event={"ID":"6d47014e-b95a-4990-8ef0-eabb2dd738a1","Type":"ContainerStarted","Data":"c49d1c8dcaff23dca85ab57ea9d775829743336a352dfc2051d2f8b0d28855af"}
Oct 30 01:00:03.666508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:03.666484    4576 scope.go:122] "RemoveContainer" containerID="fe236289769223f7aa067692a09f4e968f053cd79079dd307350d89236c06cff"
Oct 30 01:00:03.666848 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:03.666827    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5_odh-ai-gateway-infra(b987bdd2-bc91-4ac0-81b7-814a273ecb52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52"
Oct 30 01:00:03.668517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:03.668489    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" event={"ID":"24b711ee-52f3-4747-b43c-4e68af0a6250","Type":"ContainerStarted","Data":"fb7a328443209a18bc305fd47633cc89de9c96d6e080aac37c8b019213f42fd9"}
Oct 30 01:00:03.671258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:03.671100    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" event={"ID":"275f57a2-15c2-484d-8175-b35b6bf2930d","Type":"ContainerStarted","Data":"2c5125d5d7b267670feab6e44a96d9bd84f60a051e5adceaa3d361c461dbdd95"}
Oct 30 01:00:04.454534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.454493    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:04.680328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.680291    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerID="c49d1c8dcaff23dca85ab57ea9d775829743336a352dfc2051d2f8b0d28855af" exitCode=60
Oct 30 01:00:04.680328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.680324    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" event={"ID":"6d47014e-b95a-4990-8ef0-eabb2dd738a1","Type":"ContainerDied","Data":"c49d1c8dcaff23dca85ab57ea9d775829743336a352dfc2051d2f8b0d28855af"}
Oct 30 01:00:04.681399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.680398    4576 scope.go:122] "RemoveContainer" containerID="7da62b527b6f641d9843dc500339997a7ce8f907b954a6333284d6e682a4c69b"
Oct 30 01:00:04.681399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.680682    4576 scope.go:122] "RemoveContainer" containerID="c49d1c8dcaff23dca85ab57ea9d775829743336a352dfc2051d2f8b0d28855af"
Oct 30 01:00:04.681399 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:04.681029    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd_odh-ai-gateway-infra(6d47014e-b95a-4990-8ef0-eabb2dd738a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1"
Oct 30 01:00:04.682617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.682596    4576 generic.go:356] "Generic (PLEG): container finished" podID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerID="fb7a328443209a18bc305fd47633cc89de9c96d6e080aac37c8b019213f42fd9" exitCode=60
Oct 30 01:00:04.682716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.682630    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" event={"ID":"24b711ee-52f3-4747-b43c-4e68af0a6250","Type":"ContainerDied","Data":"fb7a328443209a18bc305fd47633cc89de9c96d6e080aac37c8b019213f42fd9"}
Oct 30 01:00:04.682938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.682923    4576 scope.go:122] "RemoveContainer" containerID="fb7a328443209a18bc305fd47633cc89de9c96d6e080aac37c8b019213f42fd9"
Oct 30 01:00:04.683213 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:04.683193    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49_odh-ai-gateway-infra(24b711ee-52f3-4747-b43c-4e68af0a6250)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250"
Oct 30 01:00:04.684496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.684477    4576 generic.go:356] "Generic (PLEG): container finished" podID="275f57a2-15c2-484d-8175-b35b6bf2930d" containerID="2c5125d5d7b267670feab6e44a96d9bd84f60a051e5adceaa3d361c461dbdd95" exitCode=60
Oct 30 01:00:04.684606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.684499    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" event={"ID":"275f57a2-15c2-484d-8175-b35b6bf2930d","Type":"ContainerDied","Data":"2c5125d5d7b267670feab6e44a96d9bd84f60a051e5adceaa3d361c461dbdd95"}
Oct 30 01:00:04.684770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.684750    4576 scope.go:122] "RemoveContainer" containerID="2c5125d5d7b267670feab6e44a96d9bd84f60a051e5adceaa3d361c461dbdd95"
Oct 30 01:00:04.685016 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:04.684997    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82_odh-ai-gateway-infra(275f57a2-15c2-484d-8175-b35b6bf2930d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d"
Oct 30 01:00:04.692726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.692709    4576 scope.go:122] "RemoveContainer" containerID="5bd2ade663013a3dcf6ed8ed93949b496997297d46ca0fbf10ec203da79c7d68"
Oct 30 01:00:04.705149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.705131    4576 scope.go:122] "RemoveContainer" containerID="74ebbcf36d84351d9a879800f48f38ea27033c88d0d898d544ef03e8df954c71"
Oct 30 01:00:04.886889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:04.886853    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:05.693948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:05.693920    4576 scope.go:122] "RemoveContainer" containerID="c49d1c8dcaff23dca85ab57ea9d775829743336a352dfc2051d2f8b0d28855af"
Oct 30 01:00:05.694476 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:05.694263    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd_odh-ai-gateway-infra(6d47014e-b95a-4990-8ef0-eabb2dd738a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1"
Oct 30 01:00:05.695513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:05.695491    4576 scope.go:122] "RemoveContainer" containerID="fb7a328443209a18bc305fd47633cc89de9c96d6e080aac37c8b019213f42fd9"
Oct 30 01:00:05.695780 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:05.695756    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49_odh-ai-gateway-infra(24b711ee-52f3-4747-b43c-4e68af0a6250)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250"
Oct 30 01:00:05.696986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:05.696968    4576 scope.go:122] "RemoveContainer" containerID="2c5125d5d7b267670feab6e44a96d9bd84f60a051e5adceaa3d361c461dbdd95"
Oct 30 01:00:05.697197 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:05.697180    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82_odh-ai-gateway-infra(275f57a2-15c2-484d-8175-b35b6bf2930d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d"
Oct 30 01:00:06.337466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:06.337432    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:06.983940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:06.983902    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"]
Oct 30 01:00:07.017129 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:07.017101    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"]
Oct 30 01:00:07.149108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:07.149083    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:07.265422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:07.265327    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:07.376652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:07.376619    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:07.636317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:07.636214    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:07.838137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:07.838102    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:08.183275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.181679    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 01:00:08.215808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.215727    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91dc086b-92dc-48ac-a0e1-59354998ab33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91dc086b-92dc-48ac-a0e1-59354998ab33-istiod-ca-cert\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.215959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.215824    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-podinfo\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.215959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.215855    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-data\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.215959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.215923    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-token\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-token\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.216103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.215965    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-certs\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.216103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216005    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-envoy\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.216103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216052    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-kube-api-access-8b4n6\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-kube-api-access-8b4n6\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.216281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216110    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-credential-socket\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.216281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216180    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-socket\") pod \"91dc086b-92dc-48ac-a0e1-59354998ab33\" (UID: \"91dc086b-92dc-48ac-a0e1-59354998ab33\") "
Oct 30 01:00:08.216632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216102    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91dc086b-92dc-48ac-a0e1-59354998ab33-istiod-ca-cert" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:08.216632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216473    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-credential-socket" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:08.216632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216489    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-data" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:08.216632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216431    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-certs" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:08.216632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.216623    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-socket" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:08.217182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.217165    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.217270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.217190    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.217270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.217209    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.217270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.217223    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91dc086b-92dc-48ac-a0e1-59354998ab33-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.217270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.217253    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.218691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.218662    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-kube-api-access-8b4n6" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "kube-api-access-8b4n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:08.218691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.218663    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-podinfo" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:00:08.218874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.218834    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-token" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:08.218942 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.218873    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-envoy" pod "91dc086b-92dc-48ac-a0e1-59354998ab33" (UID: "91dc086b-92dc-48ac-a0e1-59354998ab33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:08.318564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.318533    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.318564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.318565    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.318811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.318579    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91dc086b-92dc-48ac-a0e1-59354998ab33-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.318811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.318593    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b4n6\" (UniqueName: \"kubernetes.io/projected/91dc086b-92dc-48ac-a0e1-59354998ab33-kube-api-access-8b4n6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:08.391256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.390708    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:08.718340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.718309    4576 generic.go:356] "Generic (PLEG): container finished" podID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerID="d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992" exitCode=0
Oct 30 01:00:08.722292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.718371    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"
Oct 30 01:00:08.722292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.718388    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" event={"ID":"91dc086b-92dc-48ac-a0e1-59354998ab33","Type":"ContainerDied","Data":"d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992"}
Oct 30 01:00:08.722292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.718692    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b" event={"ID":"91dc086b-92dc-48ac-a0e1-59354998ab33","Type":"ContainerDied","Data":"81665df1c1e12b759fdd0c1937a911df8dfc999ae0028c8331b6d81634c4cc19"}
Oct 30 01:00:08.722292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.718719    4576 scope.go:122] "RemoveContainer" containerID="d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992"
Oct 30 01:00:08.735664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.735649    4576 scope.go:122] "RemoveContainer" containerID="d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992"
Oct 30 01:00:08.735958 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:08.735938    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992\": container with ID starting with d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992 not found: ID does not exist" containerID="d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992"
Oct 30 01:00:08.736037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.735969    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992"} err="failed to get container status \"d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992\": rpc error: code = NotFound desc = could not find container \"d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992\": container with ID starting with d1f39b3bea4b8f344661dd843f33dec149db69c2348c0309a00a8fe11b3d9992 not found: ID does not exist"
Oct 30 01:00:08.797794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.797752    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"]
Oct 30 01:00:08.800732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:08.800712    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-fjp5b"]
Oct 30 01:00:09.508936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:09.508903    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:09.628899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:09.628868    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:10.117547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:10.117513    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" path="/var/lib/kubelet/pods/91dc086b-92dc-48ac-a0e1-59354998ab33/volumes"
Oct 30 01:00:10.362404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:10.362370    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:10.631580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:10.631544    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:11.064545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:11.064507    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:11.514304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:11.514091    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:11.727750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:11.727717    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:11.949526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:11.949494    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:11.957505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:11.957464    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"]
Oct 30 01:00:11.997550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:11.992777    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md" podUID="e071f5bc-95c2-4de2-b4d7-23aae2fb53df" containerName="payload-processing" containerID="cri-o://613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113" gracePeriod=25
Oct 30 01:00:12.022688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.018336    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs" podUID="04064695-2bba-44c9-a11c-82ba8afd80fb" containerName="payload-pre-processing" containerID="cri-o://bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce" gracePeriod=25
Oct 30 01:00:12.460909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.460594    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:12.538127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.538053    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:12.565975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.565941    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275f57a2-15c2-484d-8175-b35b6bf2930d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/275f57a2-15c2-484d-8175-b35b6bf2930d-maas-api-service-ca\") pod \"275f57a2-15c2-484d-8175-b35b6bf2930d\" (UID: \"275f57a2-15c2-484d-8175-b35b6bf2930d\") "
Oct 30 01:00:12.566613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.566516    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275f57a2-15c2-484d-8175-b35b6bf2930d-maas-api-service-ca" pod "275f57a2-15c2-484d-8175-b35b6bf2930d" (UID: "275f57a2-15c2-484d-8175-b35b6bf2930d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:12.566774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.566754    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275f57a2-15c2-484d-8175-b35b6bf2930d-kube-api-access-7vgn9\" (UniqueName: \"kubernetes.io/projected/275f57a2-15c2-484d-8175-b35b6bf2930d-kube-api-access-7vgn9\") pod \"275f57a2-15c2-484d-8175-b35b6bf2930d\" (UID: \"275f57a2-15c2-484d-8175-b35b6bf2930d\") "
Oct 30 01:00:12.567373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.567349    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/275f57a2-15c2-484d-8175-b35b6bf2930d-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:12.570319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.570272    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275f57a2-15c2-484d-8175-b35b6bf2930d-kube-api-access-7vgn9" pod "275f57a2-15c2-484d-8175-b35b6bf2930d" (UID: "275f57a2-15c2-484d-8175-b35b6bf2930d"). InnerVolumeSpecName "kube-api-access-7vgn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:12.625903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.625874    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 01:00:12.701360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.701322    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgn9\" (UniqueName: \"kubernetes.io/projected/275f57a2-15c2-484d-8175-b35b6bf2930d-kube-api-access-7vgn9\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:12.760943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.760905    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:12.761386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.761077    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 01:00:12.807289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.807259    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04064695-2bba-44c9-a11c-82ba8afd80fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04064695-2bba-44c9-a11c-82ba8afd80fb-plugins-config-volume\") pod \"04064695-2bba-44c9-a11c-82ba8afd80fb\" (UID: \"04064695-2bba-44c9-a11c-82ba8afd80fb\") "
Oct 30 01:00:12.807481 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.807342    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04064695-2bba-44c9-a11c-82ba8afd80fb-kube-api-access-dpjbr\" (UniqueName: \"kubernetes.io/projected/04064695-2bba-44c9-a11c-82ba8afd80fb-kube-api-access-dpjbr\") pod \"04064695-2bba-44c9-a11c-82ba8afd80fb\" (UID: \"04064695-2bba-44c9-a11c-82ba8afd80fb\") "
Oct 30 01:00:12.807771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.807743    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04064695-2bba-44c9-a11c-82ba8afd80fb-plugins-config-volume" pod "04064695-2bba-44c9-a11c-82ba8afd80fb" (UID: "04064695-2bba-44c9-a11c-82ba8afd80fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:12.809760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.809736    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04064695-2bba-44c9-a11c-82ba8afd80fb-kube-api-access-dpjbr" pod "04064695-2bba-44c9-a11c-82ba8afd80fb" (UID: "04064695-2bba-44c9-a11c-82ba8afd80fb"). InnerVolumeSpecName "kube-api-access-dpjbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:12.897319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.896432    4576 generic.go:356] "Generic (PLEG): container finished" podID="e071f5bc-95c2-4de2-b4d7-23aae2fb53df" containerID="613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113" exitCode=0
Oct 30 01:00:12.897319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.896541    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md" event={"ID":"e071f5bc-95c2-4de2-b4d7-23aae2fb53df","Type":"ContainerDied","Data":"613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113"}
Oct 30 01:00:12.897319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.896595    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md" event={"ID":"e071f5bc-95c2-4de2-b4d7-23aae2fb53df","Type":"ContainerDied","Data":"16c1c0edb792f87d1a4966c938d172d23966b4aea3f8a1ffbb5cc6cdf933a05a"}
Oct 30 01:00:12.897319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.896625    4576 scope.go:122] "RemoveContainer" containerID="613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113"
Oct 30 01:00:12.897319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.896781    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"
Oct 30 01:00:12.906330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.906289    4576 generic.go:356] "Generic (PLEG): container finished" podID="04064695-2bba-44c9-a11c-82ba8afd80fb" containerID="bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce" exitCode=0
Oct 30 01:00:12.906679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.906656    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs" event={"ID":"04064695-2bba-44c9-a11c-82ba8afd80fb","Type":"ContainerDied","Data":"bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce"}
Oct 30 01:00:12.906856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.906839    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs" event={"ID":"04064695-2bba-44c9-a11c-82ba8afd80fb","Type":"ContainerDied","Data":"a5cf33193abbac0b7eacc407c3bb0a2fda867ae939ca14c2cf146c35fc219564"}
Oct 30 01:00:12.907042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.907026    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"
Oct 30 01:00:12.925604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.918802    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-plugins-config-volume\") pod \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\" (UID: \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\") "
Oct 30 01:00:12.926569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.926531    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-kube-api-access-ntj4c\" (UniqueName: \"kubernetes.io/projected/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-kube-api-access-ntj4c\") pod \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\" (UID: \"e071f5bc-95c2-4de2-b4d7-23aae2fb53df\") "
Oct 30 01:00:12.927515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.927489    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04064695-2bba-44c9-a11c-82ba8afd80fb-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:12.927659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.927644    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpjbr\" (UniqueName: \"kubernetes.io/projected/04064695-2bba-44c9-a11c-82ba8afd80fb-kube-api-access-dpjbr\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:12.938148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.937271    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-plugins-config-volume" pod "e071f5bc-95c2-4de2-b4d7-23aae2fb53df" (UID: "e071f5bc-95c2-4de2-b4d7-23aae2fb53df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:12.939674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.939580    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:12.966273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.956150    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82" event={"ID":"275f57a2-15c2-484d-8175-b35b6bf2930d","Type":"ContainerDied","Data":"e756cd615981d7f0d12208b7afed5f8430e247e8d852a91bb9c14d67fc8ba483"}
Oct 30 01:00:12.966273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.956330    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"
Oct 30 01:00:12.968480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.968448    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"]
Oct 30 01:00:12.968898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:12.968871    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72" podUID="ff3cb575-2086-46cc-97c4-648c170bf98e" containerName="istio-proxy" containerID="cri-o://539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186" gracePeriod=30
Oct 30 01:00:13.048256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.044117    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.069756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.068613    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:13.112609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.105458    4576 scope.go:122] "RemoveContainer" containerID="613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113"
Oct 30 01:00:13.112609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.108926    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-kube-api-access-ntj4c" pod "e071f5bc-95c2-4de2-b4d7-23aae2fb53df" (UID: "e071f5bc-95c2-4de2-b4d7-23aae2fb53df"). InnerVolumeSpecName "kube-api-access-ntj4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:13.132307 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:13.113053    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113\": container with ID starting with 613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113 not found: ID does not exist" containerID="613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113"
Oct 30 01:00:13.132307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.113113    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113"} err="failed to get container status \"613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113\": rpc error: code = NotFound desc = could not find container \"613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113\": container with ID starting with 613f783757843b5dfc1f3256492801080877d59152890ba0da91181793ce2113 not found: ID does not exist"
Oct 30 01:00:13.132307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.113158    4576 scope.go:122] "RemoveContainer" containerID="bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce"
Oct 30 01:00:13.147224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.147190    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntj4c\" (UniqueName: \"kubernetes.io/projected/e071f5bc-95c2-4de2-b4d7-23aae2fb53df-kube-api-access-ntj4c\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:13.168724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.168393    4576 scope.go:122] "RemoveContainer" containerID="bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce"
Oct 30 01:00:13.169126 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:13.168999    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce\": container with ID starting with bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce not found: ID does not exist" containerID="bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce"
Oct 30 01:00:13.169126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.169037    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce"} err="failed to get container status \"bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce\": rpc error: code = NotFound desc = could not find container \"bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce\": container with ID starting with bc7630f83eee3d197e49365c6c9179a55807749ea06afa3ac1acc86855b434ce not found: ID does not exist"
Oct 30 01:00:13.169126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.169063    4576 scope.go:122] "RemoveContainer" containerID="2c5125d5d7b267670feab6e44a96d9bd84f60a051e5adceaa3d361c461dbdd95"
Oct 30 01:00:13.230341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.230299    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"]
Oct 30 01:00:13.236109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.236058    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-95b31f79-85c4766c87-rvxhs"]
Oct 30 01:00:13.248157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.248132    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"]
Oct 30 01:00:13.253825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.253792    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-95b31f79-29888700-n8c82"]
Oct 30 01:00:13.341254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.337578    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"]
Oct 30 01:00:13.362127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.361418    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-95b31f79-79bfc65bd-699md"]
Oct 30 01:00:13.887783 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:13.887670    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:14.019256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.019082    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:14.101295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.101192    4576 scope.go:122] "RemoveContainer" containerID="fe236289769223f7aa067692a09f4e968f053cd79079dd307350d89236c06cff"
Oct 30 01:00:14.101495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.101402    4576 scope.go:122] "RemoveContainer" containerID="fb7a328443209a18bc305fd47633cc89de9c96d6e080aac37c8b019213f42fd9"
Oct 30 01:00:14.129601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.129518    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04064695-2bba-44c9-a11c-82ba8afd80fb" path="/var/lib/kubelet/pods/04064695-2bba-44c9-a11c-82ba8afd80fb/volumes"
Oct 30 01:00:14.131214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.130182    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d" path="/var/lib/kubelet/pods/275f57a2-15c2-484d-8175-b35b6bf2930d/volumes"
Oct 30 01:00:14.131214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.130727    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e071f5bc-95c2-4de2-b4d7-23aae2fb53df" path="/var/lib/kubelet/pods/e071f5bc-95c2-4de2-b4d7-23aae2fb53df/volumes"
Oct 30 01:00:14.292811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.292776    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:14.978305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.978269    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" event={"ID":"b987bdd2-bc91-4ac0-81b7-814a273ecb52","Type":"ContainerStarted","Data":"a7695a111d4f074970c24d3bd51b2c94babe24ab6fe28832ba428983cf066ceb"}
Oct 30 01:00:14.980468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:14.980439    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" event={"ID":"24b711ee-52f3-4747-b43c-4e68af0a6250","Type":"ContainerStarted","Data":"244b2339335db2cd1480ffd6c28477fe7151d2fe5cdff65e64790a9951b03336"}
Oct 30 01:00:15.020709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.020654    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" podStartSLOduration=15.020641273 podStartE2EDuration="15.020641273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:15.019610399 +0000 UTC m=+1295.090934233" watchObservedRunningTime="2026-10-30 01:00:15.020641273 +0000 UTC m=+1295.091965109"
Oct 30 01:00:15.253422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.253323    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:15.988637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.988599    4576 generic.go:356] "Generic (PLEG): container finished" podID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerID="a7695a111d4f074970c24d3bd51b2c94babe24ab6fe28832ba428983cf066ceb" exitCode=60
Oct 30 01:00:15.989145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.988662    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" event={"ID":"b987bdd2-bc91-4ac0-81b7-814a273ecb52","Type":"ContainerDied","Data":"a7695a111d4f074970c24d3bd51b2c94babe24ab6fe28832ba428983cf066ceb"}
Oct 30 01:00:15.989145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.988731    4576 scope.go:122] "RemoveContainer" containerID="fe236289769223f7aa067692a09f4e968f053cd79079dd307350d89236c06cff"
Oct 30 01:00:15.989145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.989143    4576 scope.go:122] "RemoveContainer" containerID="a7695a111d4f074970c24d3bd51b2c94babe24ab6fe28832ba428983cf066ceb"
Oct 30 01:00:15.989543 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:15.989514    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5_odh-ai-gateway-infra(b987bdd2-bc91-4ac0-81b7-814a273ecb52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52"
Oct 30 01:00:15.990852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.990833    4576 generic.go:356] "Generic (PLEG): container finished" podID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerID="244b2339335db2cd1480ffd6c28477fe7151d2fe5cdff65e64790a9951b03336" exitCode=60
Oct 30 01:00:15.990972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.990883    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" event={"ID":"24b711ee-52f3-4747-b43c-4e68af0a6250","Type":"ContainerDied","Data":"244b2339335db2cd1480ffd6c28477fe7151d2fe5cdff65e64790a9951b03336"}
Oct 30 01:00:15.991218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:15.991200    4576 scope.go:122] "RemoveContainer" containerID="244b2339335db2cd1480ffd6c28477fe7151d2fe5cdff65e64790a9951b03336"
Oct 30 01:00:15.991522 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:15.991502    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49_odh-ai-gateway-infra(24b711ee-52f3-4747-b43c-4e68af0a6250)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250"
Oct 30 01:00:16.002270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:16.002246    4576 scope.go:122] "RemoveContainer" containerID="fb7a328443209a18bc305fd47633cc89de9c96d6e080aac37c8b019213f42fd9"
Oct 30 01:00:16.030116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:16.030089    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"]
Oct 30 01:00:16.069865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:16.069832    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"]
Oct 30 01:00:16.102497 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:16.102459    4576 scope.go:122] "RemoveContainer" containerID="c49d1c8dcaff23dca85ab57ea9d775829743336a352dfc2051d2f8b0d28855af"
Oct 30 01:00:16.266713 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:16.266634    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:16.394183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:16.394146    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:17.009877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.009838    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerID="843045b036e06a46ad42761253645cc19ae3be0f0092fb95645bccbc0c89dd9e" exitCode=60
Oct 30 01:00:17.010610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.010579    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" event={"ID":"6d47014e-b95a-4990-8ef0-eabb2dd738a1","Type":"ContainerDied","Data":"843045b036e06a46ad42761253645cc19ae3be0f0092fb95645bccbc0c89dd9e"}
Oct 30 01:00:17.011209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.011181    4576 scope.go:122] "RemoveContainer" containerID="c49d1c8dcaff23dca85ab57ea9d775829743336a352dfc2051d2f8b0d28855af"
Oct 30 01:00:17.012750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.012726    4576 scope.go:122] "RemoveContainer" containerID="843045b036e06a46ad42761253645cc19ae3be0f0092fb95645bccbc0c89dd9e"
Oct 30 01:00:17.013268 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:17.013203    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd_odh-ai-gateway-infra(6d47014e-b95a-4990-8ef0-eabb2dd738a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1"
Oct 30 01:00:17.047774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.047737    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:17.258288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.258255    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:17.262436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.262416    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:17.321936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.321906    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b711ee-52f3-4747-b43c-4e68af0a6250-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b711ee-52f3-4747-b43c-4e68af0a6250-maas-api-service-ca\") pod \"24b711ee-52f3-4747-b43c-4e68af0a6250\" (UID: \"24b711ee-52f3-4747-b43c-4e68af0a6250\") "
Oct 30 01:00:17.323268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.322171    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b987bdd2-bc91-4ac0-81b7-814a273ecb52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b987bdd2-bc91-4ac0-81b7-814a273ecb52-maas-api-service-ca\") pod \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\" (UID: \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\") "
Oct 30 01:00:17.323268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.322211    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b987bdd2-bc91-4ac0-81b7-814a273ecb52-kube-api-access-dx95w\" (UniqueName: \"kubernetes.io/projected/b987bdd2-bc91-4ac0-81b7-814a273ecb52-kube-api-access-dx95w\") pod \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\" (UID: \"b987bdd2-bc91-4ac0-81b7-814a273ecb52\") "
Oct 30 01:00:17.323268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.322315    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b711ee-52f3-4747-b43c-4e68af0a6250-kube-api-access-59xw6\" (UniqueName: \"kubernetes.io/projected/24b711ee-52f3-4747-b43c-4e68af0a6250-kube-api-access-59xw6\") pod \"24b711ee-52f3-4747-b43c-4e68af0a6250\" (UID: \"24b711ee-52f3-4747-b43c-4e68af0a6250\") "
Oct 30 01:00:17.323268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.322378    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b711ee-52f3-4747-b43c-4e68af0a6250-maas-api-service-ca" pod "24b711ee-52f3-4747-b43c-4e68af0a6250" (UID: "24b711ee-52f3-4747-b43c-4e68af0a6250"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:17.323268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.322753    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b987bdd2-bc91-4ac0-81b7-814a273ecb52-maas-api-service-ca" pod "b987bdd2-bc91-4ac0-81b7-814a273ecb52" (UID: "b987bdd2-bc91-4ac0-81b7-814a273ecb52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:17.323268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.322933    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b987bdd2-bc91-4ac0-81b7-814a273ecb52-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.323268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.322953    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b711ee-52f3-4747-b43c-4e68af0a6250-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.326202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.326175    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b987bdd2-bc91-4ac0-81b7-814a273ecb52-kube-api-access-dx95w" pod "b987bdd2-bc91-4ac0-81b7-814a273ecb52" (UID: "b987bdd2-bc91-4ac0-81b7-814a273ecb52"). InnerVolumeSpecName "kube-api-access-dx95w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:17.326814 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.326791    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b711ee-52f3-4747-b43c-4e68af0a6250-kube-api-access-59xw6" pod "24b711ee-52f3-4747-b43c-4e68af0a6250" (UID: "24b711ee-52f3-4747-b43c-4e68af0a6250"). InnerVolumeSpecName "kube-api-access-59xw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:17.423939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.423911    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx95w\" (UniqueName: \"kubernetes.io/projected/b987bdd2-bc91-4ac0-81b7-814a273ecb52-kube-api-access-dx95w\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.423939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.423938    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59xw6\" (UniqueName: \"kubernetes.io/projected/24b711ee-52f3-4747-b43c-4e68af0a6250-kube-api-access-59xw6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.502771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.502750    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 01:00:17.524610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524579    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3cb575-2086-46cc-97c4-648c170bf98e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cb575-2086-46cc-97c4-648c170bf98e-istiod-ca-cert\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.524745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524627    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-socket\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.524745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524676    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-envoy\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.524745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524704    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-token\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.524745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524736    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-data\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.524984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524759    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-kube-api-access-p485z\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-kube-api-access-p485z\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.524984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524788    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-credential-socket\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.524984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524906    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-socket" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:17.524984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.524981    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3cb575-2086-46cc-97c4-648c170bf98e-istiod-ca-cert" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:17.525193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.525143    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-credential-socket" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:17.525279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.525202    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-certs\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.525335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.525302    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-podinfo\") pod \"ff3cb575-2086-46cc-97c4-648c170bf98e\" (UID: \"ff3cb575-2086-46cc-97c4-648c170bf98e\") "
Oct 30 01:00:17.525409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.525340    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-data" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:17.525484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.525450    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-certs" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:17.526184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.526159    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.526340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.526189    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.526340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.526204    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cb575-2086-46cc-97c4-648c170bf98e-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.526340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.526217    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.526340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.526262    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.527381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.527359    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-envoy" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:00:17.527491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.527379    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-token" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:17.527807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.527645    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-podinfo" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:00:17.527807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.527739    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-kube-api-access-p485z" pod "ff3cb575-2086-46cc-97c4-648c170bf98e" (UID: "ff3cb575-2086-46cc-97c4-648c170bf98e"). InnerVolumeSpecName "kube-api-access-p485z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:17.627434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.627401    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.627434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.627425    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.627434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.627436    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p485z\" (UniqueName: \"kubernetes.io/projected/ff3cb575-2086-46cc-97c4-648c170bf98e-kube-api-access-p485z\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.627722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.627451    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cb575-2086-46cc-97c4-648c170bf98e-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.885193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885158    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"]
Oct 30 01:00:17.885763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885744    4576 cpu_manager.go:414] "containerMap: removing container" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy"
Oct 30 01:00:17.885763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885762    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885771    4576 cpu_manager.go:414] "containerMap: removing container" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885776    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885803    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e071f5bc-95c2-4de2-b4d7-23aae2fb53df" containerName="payload-processing"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885811    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e071f5bc-95c2-4de2-b4d7-23aae2fb53df" containerName="payload-processing"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885822    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885829    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885841    4576 cpu_manager.go:414] "containerMap: removing container" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885850    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885860    4576 cpu_manager.go:414] "containerMap: removing container" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885864    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885874    4576 cpu_manager.go:414] "containerMap: removing container" podUID="04064695-2bba-44c9-a11c-82ba8afd80fb" containerName="payload-pre-processing"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885882    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04064695-2bba-44c9-a11c-82ba8afd80fb" containerName="payload-pre-processing"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885899    4576 cpu_manager.go:414] "containerMap: removing container" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885909    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.885912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885917    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885922    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885940    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ff3cb575-2086-46cc-97c4-648c170bf98e" containerName="istio-proxy"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.885945    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3cb575-2086-46cc-97c4-648c170bf98e" containerName="istio-proxy"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886034    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3cb575-2086-46cc-97c4-648c170bf98e" containerName="istio-proxy"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886041    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="91dc086b-92dc-48ac-a0e1-59354998ab33" containerName="istio-proxy"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886047    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d" containerName="cleanup"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886054    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e071f5bc-95c2-4de2-b4d7-23aae2fb53df" containerName="payload-processing"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886060    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886066    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886075    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886083    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886091    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="04064695-2bba-44c9-a11c-82ba8afd80fb" containerName="payload-pre-processing"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886097    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" containerName="cleanup"
Oct 30 01:00:17.886556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.886101    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.889969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.889944    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.895507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.895474    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"]
Oct 30 01:00:17.896269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.896214    4576 cpu_manager.go:414] "containerMap: removing container" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d" containerName="cleanup"
Oct 30 01:00:17.896269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.896251    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d" containerName="cleanup"
Oct 30 01:00:17.896468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.896301    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.896468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.896310    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" containerName="cleanup"
Oct 30 01:00:17.896468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.896436    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="275f57a2-15c2-484d-8175-b35b6bf2930d" containerName="cleanup"
Oct 30 01:00:17.900676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.900656    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.915148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.915110    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"]
Oct 30 01:00:17.929052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.929026    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"]
Oct 30 01:00:17.931297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931246    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.931424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931297    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.931424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931364    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.931424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931419    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpgn\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-kube-api-access-rcpgn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.931591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931469    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.931591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931546    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.931591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931582    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.931739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931609    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.931739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931652    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.931739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931683    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.931739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931718    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.931932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931752    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.931932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931798    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.931932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931829    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.931932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931856    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.931932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931921    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:17.932166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931960    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.932166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.931997    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g64t\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-kube-api-access-4g64t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:17.989726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:17.989688    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:18.026517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.026475    4576 generic.go:356] "Generic (PLEG): container finished" podID="ff3cb575-2086-46cc-97c4-648c170bf98e" containerID="539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186" exitCode=0
Oct 30 01:00:18.026994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.026521    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"
Oct 30 01:00:18.026994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.026555    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72" event={"ID":"ff3cb575-2086-46cc-97c4-648c170bf98e","Type":"ContainerDied","Data":"539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186"}
Oct 30 01:00:18.026994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.026622    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72" event={"ID":"ff3cb575-2086-46cc-97c4-648c170bf98e","Type":"ContainerDied","Data":"2e1939f07dbfb55ef606b3735eeea69cc71a3c5543c36bcc1b0d8f08239a3c66"}
Oct 30 01:00:18.026994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.026653    4576 scope.go:122] "RemoveContainer" containerID="539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186"
Oct 30 01:00:18.028984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.028875    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"
Oct 30 01:00:18.028984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.028875    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5" event={"ID":"b987bdd2-bc91-4ac0-81b7-814a273ecb52","Type":"ContainerDied","Data":"690c5a7112b14d64272cb81cd2c637331951dfeb2812e8ea7f5518e2f5d893f2"}
Oct 30 01:00:18.030892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.030870    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49" event={"ID":"24b711ee-52f3-4747-b43c-4e68af0a6250","Type":"ContainerDied","Data":"021a61144574a65b8157837319931477a0e52932e627b279f30c56fe84fb5a2c"}
Oct 30 01:00:18.030892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.030888    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"
Oct 30 01:00:18.032940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.032912    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpgn\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-kube-api-access-rcpgn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.033047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.032972    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.033047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033038    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.033165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033074    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.033165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033103    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033144    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033174    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033210    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.033367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033281    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033327    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033359    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.033828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033394    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033454    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.033828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033489    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033525    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g64t\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-kube-api-access-4g64t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033588    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.033828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033619    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.033828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.033685    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.035681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.034539    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.035681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.034579    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.035681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.035011    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.035681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.035038    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.035681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.035116    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.035681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.035297    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.035681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.035375    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.036283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.035851    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.036283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.036125    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.036392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.036289    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.040703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.040678    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.041465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.041331    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.042251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.042085    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.042636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.042618    4576 scope.go:122] "RemoveContainer" containerID="539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186"
Oct 30 01:00:18.043561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.043537    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.044282 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:18.044193    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186\": container with ID starting with 539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186 not found: ID does not exist" containerID="539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186"
Oct 30 01:00:18.044282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.044225    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186"} err="failed to get container status \"539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186\": rpc error: code = NotFound desc = could not find container \"539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186\": container with ID starting with 539f4c58896d22acbd5d689caa2d35c6a95681958b9619ed6fd9bf0f0b2d7186 not found: ID does not exist"
Oct 30 01:00:18.044282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.044269    4576 scope.go:122] "RemoveContainer" containerID="a7695a111d4f074970c24d3bd51b2c94babe24ab6fe28832ba428983cf066ceb"
Oct 30 01:00:18.053515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.053495    4576 scope.go:122] "RemoveContainer" containerID="244b2339335db2cd1480ffd6c28477fe7151d2fe5cdff65e64790a9951b03336"
Oct 30 01:00:18.057874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.057850    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g64t\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-kube-api-access-4g64t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.066642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.066592    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.067453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.067055    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpgn\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-kube-api-access-rcpgn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-njftz\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.067453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.067414    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zkldk\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.068378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.068355    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"]
Oct 30 01:00:18.128539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.128389    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:18.131403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.131382    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"]
Oct 30 01:00:18.131545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.131536    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-95b31f79-gw-openshift-default-c86fb74fc-8db72"]
Oct 30 01:00:18.188361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.188328    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"]
Oct 30 01:00:18.197582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.197542    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca324d-29888700-z7rf5"]
Oct 30 01:00:18.212533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.212498    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:18.223782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.223754    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"]
Oct 30 01:00:18.229265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.229213    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:18.238138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.237099    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:18.241243 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.241208    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"]
Oct 30 01:00:18.256327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.256139    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"]
Oct 30 01:00:18.257602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.257577    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4b973b\""
Oct 30 01:00:18.280320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.277667    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888700-7lv49"]
Oct 30 01:00:18.309260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.309187    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:00:18.333372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.333344    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4b973b-dockercfg-r6q8w\""
Oct 30 01:00:18.382300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.369317    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866c83a5-1772-4d9a-af4e-1d1986d4ae14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l\" (UID: \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:18.382300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.369361    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l452x\" (UniqueName: \"kubernetes.io/projected/866c83a5-1772-4d9a-af4e-1d1986d4ae14-kube-api-access-l452x\") pod \"payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l\" (UID: \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:18.397437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.397407    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:18.448391 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:18.448331    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:00:18.470185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.470117    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d47014e-b95a-4990-8ef0-eabb2dd738a1-kube-api-access-v89l2\" (UniqueName: \"kubernetes.io/projected/6d47014e-b95a-4990-8ef0-eabb2dd738a1-kube-api-access-v89l2\") pod \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\" (UID: \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\") "
Oct 30 01:00:18.470732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.470458    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d47014e-b95a-4990-8ef0-eabb2dd738a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47014e-b95a-4990-8ef0-eabb2dd738a1-maas-api-service-ca\") pod \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\" (UID: \"6d47014e-b95a-4990-8ef0-eabb2dd738a1\") "
Oct 30 01:00:18.471208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.471177    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866c83a5-1772-4d9a-af4e-1d1986d4ae14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l\" (UID: \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:18.471336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.471226    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l452x\" (UniqueName: \"kubernetes.io/projected/866c83a5-1772-4d9a-af4e-1d1986d4ae14-kube-api-access-l452x\") pod \"payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l\" (UID: \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:18.471621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.471595    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d47014e-b95a-4990-8ef0-eabb2dd738a1-maas-api-service-ca" pod "6d47014e-b95a-4990-8ef0-eabb2dd738a1" (UID: "6d47014e-b95a-4990-8ef0-eabb2dd738a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:18.472479 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.472455    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866c83a5-1772-4d9a-af4e-1d1986d4ae14-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l\" (UID: \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:18.473218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.473193    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d47014e-b95a-4990-8ef0-eabb2dd738a1-kube-api-access-v89l2" pod "6d47014e-b95a-4990-8ef0-eabb2dd738a1" (UID: "6d47014e-b95a-4990-8ef0-eabb2dd738a1"). InnerVolumeSpecName "kube-api-access-v89l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:18.496579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.496515    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l452x\" (UniqueName: \"kubernetes.io/projected/866c83a5-1772-4d9a-af4e-1d1986d4ae14-kube-api-access-l452x\") pod \"payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l\" (UID: \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:18.573186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.573152    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89l2\" (UniqueName: \"kubernetes.io/projected/6d47014e-b95a-4990-8ef0-eabb2dd738a1-kube-api-access-v89l2\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.573186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.573186    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47014e-b95a-4990-8ef0-eabb2dd738a1-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.597642 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:00:18.597602    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f90a5b9_4d16_48c1_90eb_b685a04942f3.slice/crio-f42f74954f9f58c5c194b883c8f3f088cc561f46bf532199be2966f29cdae5be WatchSource:0}: Error finding container f42f74954f9f58c5c194b883c8f3f088cc561f46bf532199be2966f29cdae5be: Status 404 returned error can't find the container with id f42f74954f9f58c5c194b883c8f3f088cc561f46bf532199be2966f29cdae5be
Oct 30 01:00:18.598293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.598263    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"]
Oct 30 01:00:18.600574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.600544    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:18.600665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.600605    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:18.600665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.600633    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:18.624904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.624869    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"]
Oct 30 01:00:18.647898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.647856    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:18.648143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.648111    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:18.648317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.648162    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:18.699791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:18.699764    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:19.041247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.040292    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"]
Oct 30 01:00:19.043464 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:00:19.043432    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866c83a5_1772_4d9a_af4e_1d1986d4ae14.slice/crio-94c99f9c145cd123983cd6c787286c3f4a14d9c514e3616aa5fe5774fa98eb0f WatchSource:0}: Error finding container 94c99f9c145cd123983cd6c787286c3f4a14d9c514e3616aa5fe5774fa98eb0f: Status 404 returned error can't find the container with id 94c99f9c145cd123983cd6c787286c3f4a14d9c514e3616aa5fe5774fa98eb0f
Oct 30 01:00:19.047187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.047160    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" event={"ID":"e5dcef84-ed5d-44df-b042-aa36913c4cd4","Type":"ContainerStarted","Data":"dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5"}
Oct 30 01:00:19.047318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.047215    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" event={"ID":"e5dcef84-ed5d-44df-b042-aa36913c4cd4","Type":"ContainerStarted","Data":"ba307b22e288282959751f2b7fe69f9b9eefde93eb6031247356692080f2cd65"}
Oct 30 01:00:19.049227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.049199    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" event={"ID":"9f90a5b9-4d16-48c1-90eb-b685a04942f3","Type":"ContainerStarted","Data":"8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1"}
Oct 30 01:00:19.049356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.049267    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" event={"ID":"9f90a5b9-4d16-48c1-90eb-b685a04942f3","Type":"ContainerStarted","Data":"f42f74954f9f58c5c194b883c8f3f088cc561f46bf532199be2966f29cdae5be"}
Oct 30 01:00:19.051337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.051311    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd" event={"ID":"6d47014e-b95a-4990-8ef0-eabb2dd738a1","Type":"ContainerDied","Data":"a5f43369d24aee70c20136bafe1f6b5c21f5b48fa60a410ea44c172aed467051"}
Oct 30 01:00:19.051458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.051372    4576 scope.go:122] "RemoveContainer" containerID="843045b036e06a46ad42761253645cc19ae3be0f0092fb95645bccbc0c89dd9e"
Oct 30 01:00:19.051458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.051362    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"
Oct 30 01:00:19.080499 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.080472    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:19.097868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.097814    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podStartSLOduration=2.097796238 podStartE2EDuration="2.097796238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:19.095813204 +0000 UTC m=+1299.167137042" watchObservedRunningTime="2026-10-30 01:00:19.097796238 +0000 UTC m=+1299.169120075"
Oct 30 01:00:19.150447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.150392    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podStartSLOduration=2.150370481 podStartE2EDuration="2.150370481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:19.142765966 +0000 UTC m=+1299.214089802" watchObservedRunningTime="2026-10-30 01:00:19.150370481 +0000 UTC m=+1299.221694312"
Oct 30 01:00:19.187914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.187854    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"]
Oct 30 01:00:19.204773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.204108    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888700-zkdwd"]
Oct 30 01:00:19.228704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.228683    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:19.228911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.228576    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 01:00:19.229068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.229048    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 01:00:19.240190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.239990    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 01:00:19.240190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.240055    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 01:00:19.240190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:19.240126    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:20.110390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.110351    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l" event={"ID":"866c83a5-1772-4d9a-af4e-1d1986d4ae14","Type":"ContainerStarted","Data":"a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90"}
Oct 30 01:00:20.110946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.110925    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:20.111120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.111100    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l" event={"ID":"866c83a5-1772-4d9a-af4e-1d1986d4ae14","Type":"ContainerStarted","Data":"94c99f9c145cd123983cd6c787286c3f4a14d9c514e3616aa5fe5774fa98eb0f"}
Oct 30 01:00:20.125224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.125185    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b711ee-52f3-4747-b43c-4e68af0a6250" path="/var/lib/kubelet/pods/24b711ee-52f3-4747-b43c-4e68af0a6250/volumes"
Oct 30 01:00:20.126155 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.126129    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" path="/var/lib/kubelet/pods/6d47014e-b95a-4990-8ef0-eabb2dd738a1/volumes"
Oct 30 01:00:20.127016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.126929    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b987bdd2-bc91-4ac0-81b7-814a273ecb52" path="/var/lib/kubelet/pods/b987bdd2-bc91-4ac0-81b7-814a273ecb52/volumes"
Oct 30 01:00:20.137159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.137127    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3cb575-2086-46cc-97c4-648c170bf98e" path="/var/lib/kubelet/pods/ff3cb575-2086-46cc-97c4-648c170bf98e/volumes"
Oct 30 01:00:20.226815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.226776    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 01:00:20.227109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.227077    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 01:00:20.237676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.237639    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 01:00:20.237930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:20.237903    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 01:00:21.216360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:21.216323    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 01:00:21.217459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:21.216920    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 01:00:21.218082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:21.218057    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:21.239620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:21.239588    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 01:00:21.239849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:21.239823    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 01:00:21.334313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:21.329754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:21.743821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:21.743787    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:21.886126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:21.879939    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:22.038314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:22.037218    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:22.215259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:22.214658    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 01:00:22.215259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:22.214716    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 01:00:22.241260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:22.240647    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 01:00:22.241260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:22.240701    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 01:00:22.961256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:22.954484    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:22.996258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:22.995592    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:23.022259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.007927    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:23.022259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.014860    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:23.101384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.101340    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 01:00:23.124777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.124738    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:23.210750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.176221    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 01:00:23.214097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.214019    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:23.359259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.337335    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 01:00:23.359259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.337403    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 01:00:23.359259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.337789    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 01:00:23.359259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.337815    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 01:00:23.858687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.858613    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:23.957456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.957394    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 01:00:23.958156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.958126    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 01:00:23.958776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.958759    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 01:00:23.999405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:23.999363    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:24.000104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.000083    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 01:00:24.026060 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.023192    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l" podStartSLOduration=6.023172506 podStartE2EDuration="6.023172506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:20.191641536 +0000 UTC m=+1300.262965372" watchObservedRunningTime="2026-10-30 01:00:24.023172506 +0000 UTC m=+1304.094496342"
Oct 30 01:00:24.157249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.155624    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:24.157249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.156608    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 01:00:24.180274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.179817    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:24.224301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.213559    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 01:00:24.224301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.213639    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 01:00:24.243278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.240369    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 01:00:24.243278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.240425    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 01:00:24.267263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.263696    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 01:00:24.359260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.358662    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 01:00:24.562478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.562452    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 01:00:24.565508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.565484    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 01:00:24.917310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:24.905047    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:25.213500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:25.213455    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 01:00:25.213707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:25.213544    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 01:00:25.237790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:25.237759    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 01:00:25.237950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:25.237817    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 01:00:26.213100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:26.213060    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 01:00:26.213542 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:26.213133    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 01:00:26.238053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:26.238014    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 01:00:26.238204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:26.238084    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 01:00:27.246260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:27.245893    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:27.356110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:27.355995    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:27.476401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:27.476373    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:27.942622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:27.942566    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:27.943302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:27.943277    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:27.944134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:27.944112    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:00:28.215811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:28.215514    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:28.215811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:28.215608    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:29.256389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:29.256352    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:29.256779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:29.256406    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:29.675225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:29.675101    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:30.309266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:30.304854    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-njftz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:30.309266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:30.304919    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:30.714134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:30.714097    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:30.719367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:30.715112    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:30.719367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:30.717335    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:00:31.172974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:31.172886    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:00:31.430914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:31.430836    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:31.541162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:31.541119    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:33.536813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:33.536784    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:33.895841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:33.895746    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:34.510954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:34.510920    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:35.184710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:35.184679    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:35.407490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:35.407462    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:36.959829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:36.959718    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:37.075067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:37.074046    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:37.198524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:37.198489    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:39.163258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:39.156776    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:40.044965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:40.044923    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:40.976735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:40.976692    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:41.082366 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:41.082322    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:42.158399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:42.158368    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:42.637576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:42.637547    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:42.748131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:42.748082    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:42.920071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:42.919971    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:44.772729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:44.772696    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:46.276456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:46.276424    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:47.076396 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:47.076355    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:47.206510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:47.206467    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:48.142429 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.142395    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:48.381739 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:00:48.381707    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:00:48.505166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.505132    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:48.867718 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.867636    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"]
Oct 30 01:00:48.868533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.868512    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:48.868607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.868536    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:48.868607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.868549    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:48.868607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.868557    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:48.868761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.868728    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:48.868761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.868739    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:48.868761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.868753    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:48.874527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.874499    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.894989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.892184    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"]
Oct 30 01:00:48.901103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.901070    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7452d0-openshift-default-dockercfg-tsnvp\""
Oct 30 01:00:48.942756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.942719    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:48.967568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.967529    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-data\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.967749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.967699    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-credential-socket\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.967835 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.967747    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-envoy\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.967835 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.967777    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-podinfo\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.968008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.967847    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-socket\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.968008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.967868    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff643c71-d2c0-42b4-897e-ca9d3b180362-istiod-ca-cert\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.968116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.968053    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jc5b\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-kube-api-access-2jc5b\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.968159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.968142    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-certs\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:48.968195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:48.968168    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-token\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.073153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.073104    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jc5b\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-kube-api-access-2jc5b\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.115477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.115444    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-certs\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.116569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.116529    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-token\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.116863 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.116842    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-data\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.117127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.117107    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-credential-socket\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.120053    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-envoy\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.120102    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-podinfo\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.120199    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-socket\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.120253    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff643c71-d2c0-42b4-897e-ca9d3b180362-istiod-ca-cert\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.074802    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.119680    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-certs\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.120825    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-data\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.121146    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-socket\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.125327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.121148    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff643c71-d2c0-42b4-897e-ca9d3b180362-istiod-ca-cert\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.161021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.136534    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-credential-socket\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.168425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.167777    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-podinfo\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.168425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.168341    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jc5b\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-kube-api-access-2jc5b\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.168640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.168556    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-envoy\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.170024 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.169981    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-token\") pod \"e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") " pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.201949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.201918    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:49.531099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.531064    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"]
Oct 30 01:00:49.532312 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:00:49.532286    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff643c71_d2c0_42b4_897e_ca9d3b180362.slice/crio-fd7548bb983c105fe5e48e9bfdc69d78b52eace93e4e2bafb5d1aaf27a186d34 WatchSource:0}: Error finding container fd7548bb983c105fe5e48e9bfdc69d78b52eace93e4e2bafb5d1aaf27a186d34: Status 404 returned error can't find the container with id fd7548bb983c105fe5e48e9bfdc69d78b52eace93e4e2bafb5d1aaf27a186d34
Oct 30 01:00:49.534884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.534842    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:49.535002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.534929    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:49.535002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.534970    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:00:49.594669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.594635    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:49.721650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.721582    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:49.758152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.758109    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" event={"ID":"ff643c71-d2c0-42b4-897e-ca9d3b180362","Type":"ContainerStarted","Data":"18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d"}
Oct 30 01:00:49.758380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.758174    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" event={"ID":"ff643c71-d2c0-42b4-897e-ca9d3b180362","Type":"ContainerStarted","Data":"fd7548bb983c105fe5e48e9bfdc69d78b52eace93e4e2bafb5d1aaf27a186d34"}
Oct 30 01:00:49.804963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:49.804843    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" podStartSLOduration=1.804826724 podStartE2EDuration="1.804826724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:49.802266494 +0000 UTC m=+1329.873590329" watchObservedRunningTime="2026-10-30 01:00:49.804826724 +0000 UTC m=+1329.876150560"
Oct 30 01:00:50.202937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:50.202906    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:50.212864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:50.212837    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:50.501367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:50.501266    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:50.765091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:50.765000    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:50.766098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:50.766076    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:00:52.415147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:52.414802    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:52.538852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:52.538817    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:53.282615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:53.282576    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:54.072772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.072742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:54.158433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.158400    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"]
Oct 30 01:00:54.158990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.158976    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:54.159043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.158992    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d47014e-b95a-4990-8ef0-eabb2dd738a1" containerName="cleanup"
Oct 30 01:00:54.162890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.162872    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:54.183975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.183947    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"]
Oct 30 01:00:54.186955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.186926    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:54.192971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.192952    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7452d0\""
Oct 30 01:00:54.194690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.194650    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7016a3-130b-497a-8cfd-d35b56208fe0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn\" (UID: \"cf7016a3-130b-497a-8cfd-d35b56208fe0\") " pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:54.194877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.194858    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td964\" (UniqueName: \"kubernetes.io/projected/cf7016a3-130b-497a-8cfd-d35b56208fe0-kube-api-access-td964\") pod \"payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn\" (UID: \"cf7016a3-130b-497a-8cfd-d35b56208fe0\") " pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:54.298468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.297012    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7016a3-130b-497a-8cfd-d35b56208fe0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn\" (UID: \"cf7016a3-130b-497a-8cfd-d35b56208fe0\") " pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:54.298468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.297129    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td964\" (UniqueName: \"kubernetes.io/projected/cf7016a3-130b-497a-8cfd-d35b56208fe0-kube-api-access-td964\") pod \"payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn\" (UID: \"cf7016a3-130b-497a-8cfd-d35b56208fe0\") " pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:54.298468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.298413    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7016a3-130b-497a-8cfd-d35b56208fe0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn\" (UID: \"cf7016a3-130b-497a-8cfd-d35b56208fe0\") " pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:54.323998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.323888    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td964\" (UniqueName: \"kubernetes.io/projected/cf7016a3-130b-497a-8cfd-d35b56208fe0-kube-api-access-td964\") pod \"payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn\" (UID: \"cf7016a3-130b-497a-8cfd-d35b56208fe0\") " pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:54.479953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.479924    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:54.789456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.789426    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"]
Oct 30 01:00:54.790951 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:00:54.790923    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7016a3_130b_497a_8cfd_d35b56208fe0.slice/crio-59d7d500498dd501d176f8d4565c43fffe098f2bc45cc80b9c6649ad101657c9 WatchSource:0}: Error finding container 59d7d500498dd501d176f8d4565c43fffe098f2bc45cc80b9c6649ad101657c9: Status 404 returned error can't find the container with id 59d7d500498dd501d176f8d4565c43fffe098f2bc45cc80b9c6649ad101657c9
Oct 30 01:00:54.924695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:54.924660    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:55.803455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:55.803422    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn" event={"ID":"cf7016a3-130b-497a-8cfd-d35b56208fe0","Type":"ContainerStarted","Data":"a2017bd2150f958a496c0dd8af8ee231038f26f5d825f23e87eb71ab24e46cf2"}
Oct 30 01:00:55.803866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:55.803472    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn" event={"ID":"cf7016a3-130b-497a-8cfd-d35b56208fe0","Type":"ContainerStarted","Data":"59d7d500498dd501d176f8d4565c43fffe098f2bc45cc80b9c6649ad101657c9"}
Oct 30 01:00:55.803866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:55.803628    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:00:55.844577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:55.844521    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn" podStartSLOduration=1.844506534 podStartE2EDuration="1.844506534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:55.844073604 +0000 UTC m=+1335.915397442" watchObservedRunningTime="2026-10-30 01:00:55.844506534 +0000 UTC m=+1335.915830369"
Oct 30 01:00:56.988521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:56.988484    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:57.595938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.595900    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"]
Oct 30 01:00:57.600104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.600082    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.618100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.618070    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"]
Oct 30 01:00:57.628275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.628247    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-869c367e\"/\"openshift-service-ca.crt\""
Oct 30 01:00:57.628417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.628323    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-869c367e\"/\"e2e-lifecycle-model-869c367e-kserve-self-signed-certs\""
Oct 30 01:00:57.629896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.629872    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-869c367e\"/\"default-dockercfg-822zt\""
Oct 30 01:00:57.629896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.629887    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-869c367e\"/\"kube-root-ca.crt\""
Oct 30 01:00:57.639248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.639206    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:00:57.653793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.653764    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tls-certs\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.653935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.653858    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-dshm\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.653935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.653892    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-model-cache\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.653935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.653922    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-home\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.654125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.653963    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tmp-dir\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.654125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.654091    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hhz\" (UniqueName: \"kubernetes.io/projected/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-kube-api-access-p2hhz\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.755257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.755168    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hhz\" (UniqueName: \"kubernetes.io/projected/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-kube-api-access-p2hhz\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.755536    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tls-certs\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.755628    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-dshm\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.755657    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-model-cache\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.755690    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-home\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.755730    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tmp-dir\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.756271    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tmp-dir\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.757197    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-model-cache\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.760793    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tls-certs\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.761043    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-home\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.763903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.763598    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-dshm\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.816268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.813520    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hhz\" (UniqueName: \"kubernetes.io/projected/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-kube-api-access-p2hhz\") pod \"e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") " pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:57.912502 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:57.912415    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:58.244947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:58.244925    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"]
Oct 30 01:00:58.245960 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:00:58.245938    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3246b13c_9f6f_4729_bcf2_4cbf8bd1e475.slice/crio-a4f8afe57125d815b8e7950caf0b87a61e1bf0c945a846e39a84174722dd2ca9 WatchSource:0}: Error finding container a4f8afe57125d815b8e7950caf0b87a61e1bf0c945a846e39a84174722dd2ca9: Status 404 returned error can't find the container with id a4f8afe57125d815b8e7950caf0b87a61e1bf0c945a846e39a84174722dd2ca9
Oct 30 01:00:58.837106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:58.837074    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99" event={"ID":"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475","Type":"ContainerStarted","Data":"bef23a8269531570a7478955c3ae1ec6fc9352c0fd5e5a28ed1f01ad1918ed51"}
Oct 30 01:00:58.837106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:58.837113    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99" event={"ID":"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475","Type":"ContainerStarted","Data":"a4f8afe57125d815b8e7950caf0b87a61e1bf0c945a846e39a84174722dd2ca9"}
Oct 30 01:00:58.878303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:58.878249    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99" podStartSLOduration=1.878214828 podStartE2EDuration="1.878214828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:58.877164061 +0000 UTC m=+1338.948487897" watchObservedRunningTime="2026-10-30 01:00:58.878214828 +0000 UTC m=+1338.949538664"
Oct 30 01:00:58.912614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:58.912565    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:00:58.968716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:00:58.968684    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:00.559741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:00.559708    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:00.583593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:00.583557    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:01:01.881325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:01.881293    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:03.074260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:03.062861    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:03.350313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:03.345183    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:04.071260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:04.070741    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:04.363319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:04.362532    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:04.488259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:04.485365    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:04.764256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:04.757213    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:04.926782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:04.920607    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 01:01:05.022474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:05.022329    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.064409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:05.064360    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:01:05.084378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:05.084330    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 01:01:05.143907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:05.143828    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 01:01:05.390312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:05.388403    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:06.814304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:06.814275    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:01:07.659678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:07.659639    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:07.912729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:07.912645    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:01:07.926904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:07.926868    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:01:08.031865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:08.031834    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:01:09.642659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:09.642622    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:09.780801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:09.780769    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:09.946030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:09.945994    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:11.279470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:11.279423    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:11.397079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:11.397041    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:12.180097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:12.180063    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:12.368152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:12.368121    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:13.033048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.033011    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:13.154554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.154521    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:13.277896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.277860    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"]
Oct 30 01:01:13.287055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.286977    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:13.305846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.305816    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b814fb-dockercfg-vsdwd\""
Oct 30 01:01:13.318978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.318945    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"]
Oct 30 01:01:13.333399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.333327    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b814fb\""
Oct 30 01:01:13.380256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.377159    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m85k\" (UniqueName: \"kubernetes.io/projected/35da3089-0e0e-468c-bed3-91f552f47818-kube-api-access-8m85k\") pod \"payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t\" (UID: \"35da3089-0e0e-468c-bed3-91f552f47818\") " pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:13.380256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.377271    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35da3089-0e0e-468c-bed3-91f552f47818-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t\" (UID: \"35da3089-0e0e-468c-bed3-91f552f47818\") " pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:13.478734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.478696    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m85k\" (UniqueName: \"kubernetes.io/projected/35da3089-0e0e-468c-bed3-91f552f47818-kube-api-access-8m85k\") pod \"payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t\" (UID: \"35da3089-0e0e-468c-bed3-91f552f47818\") " pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:13.479684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.478788    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35da3089-0e0e-468c-bed3-91f552f47818-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t\" (UID: \"35da3089-0e0e-468c-bed3-91f552f47818\") " pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:13.479791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.479757    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35da3089-0e0e-468c-bed3-91f552f47818-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t\" (UID: \"35da3089-0e0e-468c-bed3-91f552f47818\") " pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:13.499847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.499817    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m85k\" (UniqueName: \"kubernetes.io/projected/35da3089-0e0e-468c-bed3-91f552f47818-kube-api-access-8m85k\") pod \"payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t\" (UID: \"35da3089-0e0e-468c-bed3-91f552f47818\") " pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:13.624364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.624285    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:13.929087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:13.929049    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"]
Oct 30 01:01:13.929656 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:01:13.929630    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35da3089_0e0e_468c_bed3_91f552f47818.slice/crio-7eecf5b4862ee19d09010d06791c4db44041600e4b6814df3ddc4b906b7ec271 WatchSource:0}: Error finding container 7eecf5b4862ee19d09010d06791c4db44041600e4b6814df3ddc4b906b7ec271: Status 404 returned error can't find the container with id 7eecf5b4862ee19d09010d06791c4db44041600e4b6814df3ddc4b906b7ec271
Oct 30 01:01:14.064344 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:14.064316    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:14.074160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:14.074126    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t" event={"ID":"35da3089-0e0e-468c-bed3-91f552f47818","Type":"ContainerStarted","Data":"a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680"}
Oct 30 01:01:14.074342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:14.074176    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t" event={"ID":"35da3089-0e0e-468c-bed3-91f552f47818","Type":"ContainerStarted","Data":"7eecf5b4862ee19d09010d06791c4db44041600e4b6814df3ddc4b906b7ec271"}
Oct 30 01:01:14.074342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:14.074207    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:14.118124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:14.118072    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t" podStartSLOduration=1.118059419 podStartE2EDuration="1.118059419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:14.117272466 +0000 UTC m=+1354.188596301" watchObservedRunningTime="2026-10-30 01:01:14.118059419 +0000 UTC m=+1354.189383254"
Oct 30 01:01:15.615276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:15.615213    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:16.043827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.043795    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"]
Oct 30 01:01:16.050521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.050498    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.072964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.072934    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"]
Oct 30 01:01:16.081518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.081487    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 01:01:16.081661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.081562    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 01:01:16.081661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.081590    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 01:01:16.081661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.081501    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lrqvm\""
Oct 30 01:01:16.115193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.115155    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.115377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.115339    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa20c39-a9fd-4b88-a6fa-c8211884757a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.115440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.115377    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.115513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.115492    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.115580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.115564    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h4zz\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-kube-api-access-5h4zz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.115635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.115600    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.115693 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.115676    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.217491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.217451    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa20c39-a9fd-4b88-a6fa-c8211884757a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.217491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.217495    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.217763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.217553    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.217763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.217587    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h4zz\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-kube-api-access-5h4zz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.217763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.217652    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.217763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.217743    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.218006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.217974    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.218936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.218902    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.220277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.220255    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.220387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.220341    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa20c39-a9fd-4b88-a6fa-c8211884757a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.220694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.220674    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.220755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.220694    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.237754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.237731    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.238104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.238081    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h4zz\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-kube-api-access-5h4zz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5t76\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.383314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.383205    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:16.696992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:16.696966    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"]
Oct 30 01:01:16.698596 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:01:16.698572    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa20c39_a9fd_4b88_a6fa_c8211884757a.slice/crio-6422ee3606b53f628a8626b4b1ba011eab16e667c30c2740a4afe123e3f18412 WatchSource:0}: Error finding container 6422ee3606b53f628a8626b4b1ba011eab16e667c30c2740a4afe123e3f18412: Status 404 returned error can't find the container with id 6422ee3606b53f628a8626b4b1ba011eab16e667c30c2740a4afe123e3f18412
Oct 30 01:01:17.099594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:17.099505    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76" event={"ID":"faa20c39-a9fd-4b88-a6fa-c8211884757a","Type":"ContainerStarted","Data":"6422ee3606b53f628a8626b4b1ba011eab16e667c30c2740a4afe123e3f18412"}
Oct 30 01:01:17.319431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:17.319396    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:17.430838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:17.430743    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:17.535504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:17.535470    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:18.387336 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:01:18.387272    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:01:19.300797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:19.300752    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:19.760351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:19.760315    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:01:19.760760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:19.760392    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:01:20.138025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:20.137918    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76" event={"ID":"faa20c39-a9fd-4b88-a6fa-c8211884757a","Type":"ContainerStarted","Data":"3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c"}
Oct 30 01:01:20.138025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:20.137988    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:20.140016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:20.139983    4576 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c5t76 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:01:20.140156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:20.140031    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76" podUID="faa20c39-a9fd-4b88-a6fa-c8211884757a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:01:20.213656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:20.213608    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76" podStartSLOduration=1.155312209 podStartE2EDuration="4.213594018s" totalImagesPullingTime="3.058281809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:20.197857366 +0000 UTC m=+1360.269181202" watchObservedRunningTime="2026-10-30 01:01:20.213594018 +0000 UTC m=+1360.284917852"
Oct 30 01:01:20.959826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:20.959785    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:21.147120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:21.147089    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:01:22.977340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:22.977307    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:24.457569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:24.457534    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:25.083775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:25.083742    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:01:26.075072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:26.075035    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:27.868704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:27.868667    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:29.391065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:29.391035    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:31.178417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:31.178386    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:31.656638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:31.656602    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:31.767337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:31.767288    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:32.591932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:32.591893    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:33.491772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:33.491733    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:34.115156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:34.115123    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:34.712954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:34.712916    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:35.825045 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:35.825010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:35.925518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:35.925486    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:37.801954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:37.801919    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:37.911060 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:37.911014    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:38.151003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:38.150911    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:38.328458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:38.328411    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:38.478323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:38.478288    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:38.700276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:38.700046    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:38.817267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:38.817146    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:39.324917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:39.324865    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:39.426433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:39.426378    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:40.320120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:40.320084    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:42.252369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:42.252336    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:42.361549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:42.361459    4576 generic.go:356] "Generic (PLEG): container finished" podID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerID="18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d" exitCode=0
Oct 30 01:01:42.361549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:42.361534    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" event={"ID":"ff643c71-d2c0-42b4-897e-ca9d3b180362","Type":"ContainerDied","Data":"18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d"}
Oct 30 01:01:42.361953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:42.361935    4576 scope.go:122] "RemoveContainer" containerID="18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d"
Oct 30 01:01:42.363509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:42.363480    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:01:42.363641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:42.363557    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:01:42.363641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:42.363591    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:01:42.400043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:42.400010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:43.394259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:43.391784    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" event={"ID":"ff643c71-d2c0-42b4-897e-ca9d3b180362","Type":"ContainerStarted","Data":"5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4"}
Oct 30 01:01:43.967117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:43.967080    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:44.093904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:44.093863    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:44.220362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:44.210862    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:01:44.221414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:44.221368    4576 patch_prober.go:29] interesting pod/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 01:01:44.221581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:44.221458    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 01:01:44.710773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:44.710736    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:45.202847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:45.202803    4576 patch_prober.go:29] interesting pod/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 01:01:45.203022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:45.202875    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 01:01:45.989272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:45.989130    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:46.129344 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:46.129301    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:46.203094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:46.203063    4576 patch_prober.go:29] interesting pod/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 01:01:46.203332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:46.203132    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 01:01:47.020711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:47.020679    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:47.202564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:47.202531    4576 patch_prober.go:29] interesting pod/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 01:01:47.202758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:47.202597    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 01:01:48.202801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:48.202767    4576 patch_prober.go:29] interesting pod/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 01:01:48.203205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:48.202829    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 01:01:48.395410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:48.395372    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:48.404472 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:01:48.404438    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:01:49.025722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:49.025683    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:49.206157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:49.206123    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:49.208503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:49.208481    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:01:49.208917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:49.208894    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:01:49.209602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:49.209585    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:01:49.613106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:49.613075    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:49.732287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:49.732251    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:51.799771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:51.799669    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:53.803537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:53.803506    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:53.918653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:53.918619    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:55.494402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:55.494368    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:55.616981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:55.616947    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:56.373981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:56.373951    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:57.942747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:57.942714    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:01:59.571700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:01:59.571669    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:00.717795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:00.715010    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:02:01.353037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:01.352999    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:03.035261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:03.034029    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"]
Oct 30 01:02:03.049819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:03.049783    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eaca62e-sshc9"]
Oct 30 01:02:03.561259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:03.561174    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:04.123726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:04.123688    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d8a748-02fc-46c2-8b88-d67ffce24f16" path="/var/lib/kubelet/pods/65d8a748-02fc-46c2-8b88-d67ffce24f16/volumes"
Oct 30 01:02:05.501048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:05.501004    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:07.305832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:07.305802    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:09.552051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:09.552021    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:11.223648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:11.223616    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:11.337762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:11.337726    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:11.446966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:11.446936    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:12.266916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:12.266880    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:12.885011 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:12.884959    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:13.329668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:13.329633    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:13.505258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:13.499794    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:13.769005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:13.768974    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:15.800699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:15.800662    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:16.163975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:16.163887    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:17.438074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:17.438044    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:17.630646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:17.630610    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:17.750457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:17.750379    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:18.382397 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:02:18.382358    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:02:19.207027 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.206991    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"]
Oct 30 01:02:19.212610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.212582    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.229819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.229792    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"]
Oct 30 01:02:19.275563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.275532    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b814fb-openshift-default-dockercfg-mp8rv\""
Oct 30 01:02:19.275777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.275743    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:19.341458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341417    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.341606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341530    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.341673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341619    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.341673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341644    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.341673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341664    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.341783 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341688    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddvx\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-kube-api-access-mddvx\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.341841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341822    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639ecc18-2109-4983-905b-6d7ae273e375-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.341901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341886    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.341948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.341928    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639ecc18-2109-4983-905b-6d7ae273e375-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.397111 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.397073    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:19.443058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443018    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443085    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443144    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443165    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443191    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443226    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddvx\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-kube-api-access-mddvx\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443334    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639ecc18-2109-4983-905b-6d7ae273e375-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443426    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443444    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-certs\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443480    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639ecc18-2109-4983-905b-6d7ae273e375-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443524    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443633    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-credential-socket\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.443841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.443725    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-data\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.444021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.444003    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639ecc18-2109-4983-905b-6d7ae273e375-istiod-ca-cert\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.445667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.445645    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-envoy\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.445963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.445945    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639ecc18-2109-4983-905b-6d7ae273e375-istio-podinfo\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.464667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.464596    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddvx\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-kube-api-access-mddvx\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.464810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.464671    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-istio-token\") pod \"e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") " pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.530101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.530069    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:19.926107 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:02:19.926072    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639ecc18_2109_4983_905b_6d7ae273e375.slice/crio-7205f1b2c19e8ea68c6bebb2b6e11c4d3646ec12b7e03f50b61c7657eec5f7b8 WatchSource:0}: Error finding container 7205f1b2c19e8ea68c6bebb2b6e11c4d3646ec12b7e03f50b61c7657eec5f7b8: Status 404 returned error can't find the container with id 7205f1b2c19e8ea68c6bebb2b6e11c4d3646ec12b7e03f50b61c7657eec5f7b8
Oct 30 01:02:19.927224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.927195    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"]
Oct 30 01:02:19.929124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.929089    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:02:19.929260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.929171    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:02:19.929260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.929218    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:02:19.967566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:19.967500    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:20.798072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:20.798038    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" event={"ID":"639ecc18-2109-4983-905b-6d7ae273e375","Type":"ContainerStarted","Data":"89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689"}
Oct 30 01:02:20.798530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:20.798090    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" event={"ID":"639ecc18-2109-4983-905b-6d7ae273e375","Type":"ContainerStarted","Data":"7205f1b2c19e8ea68c6bebb2b6e11c4d3646ec12b7e03f50b61c7657eec5f7b8"}
Oct 30 01:02:21.532559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:21.532528    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:21.534425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:21.534400    4576 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 01:02:21.534565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:21.534448    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 01:02:22.246538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:22.246509    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:22.532852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:22.532761    4576 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 01:02:22.532852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:22.532821    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 01:02:22.598632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:22.598596    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:23.532872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:23.532828    4576 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 01:02:23.533311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:23.532889    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 01:02:24.321003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:24.320963    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:24.435917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:24.435884    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:24.532407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:24.532363    4576 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 01:02:24.532580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:24.532433    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 01:02:24.540346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:24.540318    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:25.527285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:25.524345    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:25.532806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:25.532754    4576 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 01:02:25.533024 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:25.532818    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 01:02:25.637066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:25.637031    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:26.538675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:26.538645    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:26.539120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:26.539095    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:26.540299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:26.540185    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:02:26.599302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:26.599220    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" podStartSLOduration=7.599203722 podStartE2EDuration="7.599203722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:20.890547949 +0000 UTC m=+1420.961871784" watchObservedRunningTime="2026-10-30 01:02:26.599203722 +0000 UTC m=+1426.670527557"
Oct 30 01:02:27.471652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:27.471616    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:29.277641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:29.277599    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:30.865079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:30.865039    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:32.384438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:32.384399    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:32.815251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:32.815205    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:34.079818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:34.079781    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:34.331518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:34.331434    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:35.645339 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:35.645301    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:37.027170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:37.027135    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"]
Oct 30 01:02:37.043852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:37.043816    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e51eefe-89t4h"]
Oct 30 01:02:37.184156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:37.184104    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:38.110220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:38.110179    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99256b7-7ba1-4efe-8fc0-434e0481b30d" path="/var/lib/kubelet/pods/e99256b7-7ba1-4efe-8fc0-434e0481b30d/volumes"
Oct 30 01:02:38.777823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:38.777788    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:39.388448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:39.388412    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:02:40.954772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:40.954725    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:43.986759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:43.986356    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zkldk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:43.986759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:43.986426    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:44.059934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:44.059901    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"]
Oct 30 01:02:44.079151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:44.079102    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009641b3-xfvkq"]
Oct 30 01:02:44.150566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:44.150522    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8" path="/var/lib/kubelet/pods/65a4a01b-9c4b-4265-a5c0-0fa11d7ccdc8/volumes"
Oct 30 01:02:44.781074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:44.781039    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:47.476537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:47.476499    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:48.383171 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:02:48.383127    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:02:48.527874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:48.527837    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:48.974738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:48.974705    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:49.351422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.351336    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"]
Oct 30 01:02:49.374657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.374627    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.386135 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.386105    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"]
Oct 30 01:02:49.390279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.390225    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-openshift-service-ca\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.390431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.390398    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whs26\" (UniqueName: \"kubernetes.io/projected/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-kube-api-access-whs26\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.390563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.390461    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-maas-api-tls\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.390764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.390734    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-metrics-certs\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.395277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.394480    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ee0869-serving-cert\""
Oct 30 01:02:49.465334 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.463320    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:49.497079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.497017    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-metrics-certs\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.497958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.497931    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-openshift-service-ca\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.499109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.498800    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-openshift-service-ca\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.499658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.499638    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whs26\" (UniqueName: \"kubernetes.io/projected/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-kube-api-access-whs26\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.499834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.499818    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-maas-api-tls\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.507652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.507624    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-metrics-certs\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.509458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.509420    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-maas-api-tls\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.524982 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.524947    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whs26\" (UniqueName: \"kubernetes.io/projected/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-kube-api-access-whs26\") pod \"maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:49.576981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.576944    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:49.701823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.695890    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:49.716523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:49.716490    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:50.096491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:50.096461    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"]
Oct 30 01:02:50.099555 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:02:50.099523    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a073ffb_6cf4_485d_9ca5_55ec6006e25c.slice/crio-c43ae1560751c0ab798dc6f87c7a4169036ed06b8c7c7dbe60eaf593ba302ff4 WatchSource:0}: Error finding container c43ae1560751c0ab798dc6f87c7a4169036ed06b8c7c7dbe60eaf593ba302ff4: Status 404 returned error can't find the container with id c43ae1560751c0ab798dc6f87c7a4169036ed06b8c7c7dbe60eaf593ba302ff4
Oct 30 01:02:51.101050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:51.101014    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl" event={"ID":"8a073ffb-6cf4-485d-9ca5-55ec6006e25c","Type":"ContainerStarted","Data":"1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47"}
Oct 30 01:02:51.101535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:51.101073    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl" event={"ID":"8a073ffb-6cf4-485d-9ca5-55ec6006e25c","Type":"ContainerStarted","Data":"c43ae1560751c0ab798dc6f87c7a4169036ed06b8c7c7dbe60eaf593ba302ff4"}
Oct 30 01:02:51.101535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:51.101249    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:51.166779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:51.166710    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl" podStartSLOduration=1.816174698 podStartE2EDuration="2.166691548s" totalImagesPullingTime="350.51685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:51.161343159 +0000 UTC m=+1451.232666993" watchObservedRunningTime="2026-10-30 01:02:51.166691548 +0000 UTC m=+1451.238015385"
Oct 30 01:02:51.647192 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:51.647152    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:55.542672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:55.542622    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:55.660458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:55.660425    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:55.781662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:55.781631    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:55.899502 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:55.899260    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:57.116013 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:57.115986    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:02:57.277969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:57.277936    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:02:59.549535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:02:59.549506    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:00.059084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:00.059043    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:01.898273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:01.898196    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:03.907383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:03.907352    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:05.817666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:05.817627    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:07.622691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:07.622656    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:09.348355 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:09.348317    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:09.809419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:09.809380    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:10.271164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:10.271123    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:10.738291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:10.733894    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:11.664072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:11.664028    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:12.093295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:12.093256    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"]
Oct 30 01:03:12.126629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:12.126597    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:13.128820 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:13.128784    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:13.591640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:13.591605    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:14.576991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:14.576956    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:14.714477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:14.714443    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:15.176592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:15.176553    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:03:15.776976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:15.776945    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:16.444707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.444653    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:16.582526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.582489    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:16.619210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.619176    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"]
Oct 30 01:03:16.623631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.623610    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:16.645379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.645346    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"]
Oct 30 01:03:16.653513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.653480    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-b05110\""
Oct 30 01:03:16.653513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.653499    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-b05110-dockercfg-7swsr\""
Oct 30 01:03:16.683595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.683556    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50140ed-d37d-4110-8331-71bf08574258-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff\" (UID: \"d50140ed-d37d-4110-8331-71bf08574258\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:16.683950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.683925    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb76g\" (UniqueName: \"kubernetes.io/projected/d50140ed-d37d-4110-8331-71bf08574258-kube-api-access-sb76g\") pod \"payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff\" (UID: \"d50140ed-d37d-4110-8331-71bf08574258\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:16.786099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.785997    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50140ed-d37d-4110-8331-71bf08574258-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff\" (UID: \"d50140ed-d37d-4110-8331-71bf08574258\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:16.786099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.786084    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb76g\" (UniqueName: \"kubernetes.io/projected/d50140ed-d37d-4110-8331-71bf08574258-kube-api-access-sb76g\") pod \"payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff\" (UID: \"d50140ed-d37d-4110-8331-71bf08574258\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:16.786798 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.786774    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50140ed-d37d-4110-8331-71bf08574258-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff\" (UID: \"d50140ed-d37d-4110-8331-71bf08574258\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:16.807110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.807082    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb76g\" (UniqueName: \"kubernetes.io/projected/d50140ed-d37d-4110-8331-71bf08574258-kube-api-access-sb76g\") pod \"payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff\" (UID: \"d50140ed-d37d-4110-8331-71bf08574258\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:16.940810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:16.940765    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:17.093763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.093631    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn" podUID="cf7016a3-130b-497a-8cfd-d35b56208fe0" containerName="payload-processing" containerID="cri-o://a2017bd2150f958a496c0dd8af8ee231038f26f5d825f23e87eb71ab24e46cf2" gracePeriod=25
Oct 30 01:03:17.248844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.248813    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"]
Oct 30 01:03:17.251435 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:03:17.251382    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50140ed_d37d_4110_8331_71bf08574258.slice/crio-ac9027db06352e04ae81b2d6314b8d0762097a7481897177ac9e2ce9f67a66f1 WatchSource:0}: Error finding container ac9027db06352e04ae81b2d6314b8d0762097a7481897177ac9e2ce9f67a66f1: Status 404 returned error can't find the container with id ac9027db06352e04ae81b2d6314b8d0762097a7481897177ac9e2ce9f67a66f1
Oct 30 01:03:17.339357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.335450    4576 generic.go:356] "Generic (PLEG): container finished" podID="cf7016a3-130b-497a-8cfd-d35b56208fe0" containerID="a2017bd2150f958a496c0dd8af8ee231038f26f5d825f23e87eb71ab24e46cf2" exitCode=0
Oct 30 01:03:17.339357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.335647    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn" event={"ID":"cf7016a3-130b-497a-8cfd-d35b56208fe0","Type":"ContainerDied","Data":"a2017bd2150f958a496c0dd8af8ee231038f26f5d825f23e87eb71ab24e46cf2"}
Oct 30 01:03:17.339357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.338763    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff" event={"ID":"d50140ed-d37d-4110-8331-71bf08574258","Type":"ContainerStarted","Data":"ac9027db06352e04ae81b2d6314b8d0762097a7481897177ac9e2ce9f67a66f1"}
Oct 30 01:03:17.397355 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.397325    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:03:17.501957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.501926    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7016a3-130b-497a-8cfd-d35b56208fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7016a3-130b-497a-8cfd-d35b56208fe0-plugins-config-volume\") pod \"cf7016a3-130b-497a-8cfd-d35b56208fe0\" (UID: \"cf7016a3-130b-497a-8cfd-d35b56208fe0\") "
Oct 30 01:03:17.502169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.502126    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7016a3-130b-497a-8cfd-d35b56208fe0-kube-api-access-td964\" (UniqueName: \"kubernetes.io/projected/cf7016a3-130b-497a-8cfd-d35b56208fe0-kube-api-access-td964\") pod \"cf7016a3-130b-497a-8cfd-d35b56208fe0\" (UID: \"cf7016a3-130b-497a-8cfd-d35b56208fe0\") "
Oct 30 01:03:17.502368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.502343    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7016a3-130b-497a-8cfd-d35b56208fe0-plugins-config-volume" pod "cf7016a3-130b-497a-8cfd-d35b56208fe0" (UID: "cf7016a3-130b-497a-8cfd-d35b56208fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:17.502806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.502782    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7016a3-130b-497a-8cfd-d35b56208fe0-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:17.504527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.504503    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7016a3-130b-497a-8cfd-d35b56208fe0-kube-api-access-td964" pod "cf7016a3-130b-497a-8cfd-d35b56208fe0" (UID: "cf7016a3-130b-497a-8cfd-d35b56208fe0"). InnerVolumeSpecName "kube-api-access-td964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:17.604112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.604021    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td964\" (UniqueName: \"kubernetes.io/projected/cf7016a3-130b-497a-8cfd-d35b56208fe0-kube-api-access-td964\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:17.885170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:17.885079    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:18.346351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:18.346314    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"
Oct 30 01:03:18.346351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:18.346329    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn" event={"ID":"cf7016a3-130b-497a-8cfd-d35b56208fe0","Type":"ContainerDied","Data":"59d7d500498dd501d176f8d4565c43fffe098f2bc45cc80b9c6649ad101657c9"}
Oct 30 01:03:18.346640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:18.346393    4576 scope.go:122] "RemoveContainer" containerID="a2017bd2150f958a496c0dd8af8ee231038f26f5d825f23e87eb71ab24e46cf2"
Oct 30 01:03:18.348094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:18.348054    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff" event={"ID":"d50140ed-d37d-4110-8331-71bf08574258","Type":"ContainerStarted","Data":"c93f6425dfad5b07022ec4f1a8178f7ca4998ce4f44bc13459260d866e76576a"}
Oct 30 01:03:18.348209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:18.348139    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:18.383634 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:03:18.383593    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:03:18.404178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:18.404144    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"]
Oct 30 01:03:18.417738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:18.417694    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7452d0-7bbdf4855d-24gjn"]
Oct 30 01:03:18.447641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:18.447557    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff" podStartSLOduration=2.447536756 podStartE2EDuration="2.447536756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:18.443926142 +0000 UTC m=+1478.515249979" watchObservedRunningTime="2026-10-30 01:03:18.447536756 +0000 UTC m=+1478.518860593"
Oct 30 01:03:19.326809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:19.326766    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:20.111462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:20.111420    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7016a3-130b-497a-8cfd-d35b56208fe0" path="/var/lib/kubelet/pods/cf7016a3-130b-497a-8cfd-d35b56208fe0/volumes"
Oct 30 01:03:20.114822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:20.114778    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:20.377505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:20.377417    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:20.627732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:20.627608    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:21.271836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:21.271805    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:21.386360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:21.386272    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:23.166767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:23.166735    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:23.735359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:23.735313    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:24.442766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:24.442720    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:25.366320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:25.366287    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:25.488931 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:25.488898    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:27.716372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:27.716342    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:27.844216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:27.844173    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:28.044431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:28.044317    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:29.050886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:29.050850    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:29.175113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:29.175080    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:29.361272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:29.361160    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:03:30.226989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:30.226955    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:31.763216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:31.763185    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"]
Oct 30 01:03:31.763668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:31.763407    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy" containerID="cri-o://30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578" gracePeriod=30
Oct 30 01:03:32.186508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:32.186422    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:32.950311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:32.946334    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:33.951596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:33.951563    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:34.459260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:34.459195    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:35.755295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:35.755252    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"]
Oct 30 01:03:35.915004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:35.914953    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:36.032959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:36.032871    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:36.228553 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:36.228518    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:37.095737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.095709    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 01:03:37.162414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.162369    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-podinfo\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.162694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.162444    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-envoy\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.162694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.162473    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-socket\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.162694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.162596    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istiod-ca-cert\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.162694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.162659    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-kube-api-access-ftssz\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-kube-api-access-ftssz\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.162694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.162693    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-certs\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.163010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.162735    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-data\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.163342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.163316    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-socket" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:37.164183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.164157    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-certs" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:37.164564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.164533    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istiod-ca-cert" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:37.164668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.164600    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-data" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:37.164821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.164795    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-credential-socket\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.164943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.164872    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-token\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-token\") pod \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\" (UID: \"6d71d034-e6e6-4133-be6f-a22b75bd12d9\") "
Oct 30 01:03:37.165338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.165302    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-credential-socket" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:37.166461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.166434    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-envoy" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:37.166577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.166495    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-kube-api-access-ftssz" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "kube-api-access-ftssz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:37.167547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.167522    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.167647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.167549    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftssz\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-kube-api-access-ftssz\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.167647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.167563    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.167647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.167576    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.167647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.167588    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.167647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.167600    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.167647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.167613    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d71d034-e6e6-4133-be6f-a22b75bd12d9-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.169564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.169541    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-podinfo" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:03:37.169653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.169558    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-token" pod "6d71d034-e6e6-4133-be6f-a22b75bd12d9" (UID: "6d71d034-e6e6-4133-be6f-a22b75bd12d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:37.269025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.268991    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.269025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.269020    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d71d034-e6e6-4133-be6f-a22b75bd12d9-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:37.499285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.499211    4576 generic.go:356] "Generic (PLEG): container finished" podID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerID="30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578" exitCode=0
Oct 30 01:03:37.499464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.499306    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"
Oct 30 01:03:37.499464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.499304    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" event={"ID":"6d71d034-e6e6-4133-be6f-a22b75bd12d9","Type":"ContainerDied","Data":"30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578"}
Oct 30 01:03:37.499464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.499390    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf" event={"ID":"6d71d034-e6e6-4133-be6f-a22b75bd12d9","Type":"ContainerDied","Data":"3583114197dce765666117f2510fd79a211e37be89ca340686c8738abb1ad5ab"}
Oct 30 01:03:37.499464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.499414    4576 scope.go:122] "RemoveContainer" containerID="30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578"
Oct 30 01:03:37.511006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.510987    4576 scope.go:122] "RemoveContainer" containerID="30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578"
Oct 30 01:03:37.511285 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:03:37.511268    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578\": container with ID starting with 30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578 not found: ID does not exist" containerID="30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578"
Oct 30 01:03:37.511377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.511296    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578"} err="failed to get container status \"30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578\": rpc error: code = NotFound desc = could not find container \"30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578\": container with ID starting with 30e95198064c9f3aee838c0ce498bb03197a3df2ee520681b2ee0a6814774578 not found: ID does not exist"
Oct 30 01:03:37.556156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.556123    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"]
Oct 30 01:03:37.569857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.569818    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-bxxlf"]
Oct 30 01:03:37.915298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:37.915198    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:38.113037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:38.113003    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" path="/var/lib/kubelet/pods/6d71d034-e6e6-4133-be6f-a22b75bd12d9/volumes"
Oct 30 01:03:39.575214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:39.575183    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:40.372473 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:40.372444    4576 scope.go:122] "RemoveContainer" containerID="9ca6ef38f2bcf790be3e3b0940420f1365d33c448a092726c40f944bc45ae55c"
Oct 30 01:03:40.382201 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:40.382178    4576 scope.go:122] "RemoveContainer" containerID="8d8b5b65f5ff5d1f59c0f02c9d4658c87634a411b9267966c5d647947e24236c"
Oct 30 01:03:40.756488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:40.756424    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t" podUID="35da3089-0e0e-468c-bed3-91f552f47818" containerName="payload-processing" containerID="cri-o://a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680" gracePeriod=25
Oct 30 01:03:41.076116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.076089    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:03:41.236816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.236779    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35da3089-0e0e-468c-bed3-91f552f47818-kube-api-access-8m85k\" (UniqueName: \"kubernetes.io/projected/35da3089-0e0e-468c-bed3-91f552f47818-kube-api-access-8m85k\") pod \"35da3089-0e0e-468c-bed3-91f552f47818\" (UID: \"35da3089-0e0e-468c-bed3-91f552f47818\") "
Oct 30 01:03:41.237112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.237096    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35da3089-0e0e-468c-bed3-91f552f47818-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35da3089-0e0e-468c-bed3-91f552f47818-plugins-config-volume\") pod \"35da3089-0e0e-468c-bed3-91f552f47818\" (UID: \"35da3089-0e0e-468c-bed3-91f552f47818\") "
Oct 30 01:03:41.237460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.237436    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35da3089-0e0e-468c-bed3-91f552f47818-plugins-config-volume" pod "35da3089-0e0e-468c-bed3-91f552f47818" (UID: "35da3089-0e0e-468c-bed3-91f552f47818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:41.237701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.237687    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35da3089-0e0e-468c-bed3-91f552f47818-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:41.239093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.239064    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35da3089-0e0e-468c-bed3-91f552f47818-kube-api-access-8m85k" pod "35da3089-0e0e-468c-bed3-91f552f47818" (UID: "35da3089-0e0e-468c-bed3-91f552f47818"). InnerVolumeSpecName "kube-api-access-8m85k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:41.339685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.339607    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m85k\" (UniqueName: \"kubernetes.io/projected/35da3089-0e0e-468c-bed3-91f552f47818-kube-api-access-8m85k\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:03:41.485223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.485184    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:41.530160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.530126    4576 generic.go:356] "Generic (PLEG): container finished" podID="35da3089-0e0e-468c-bed3-91f552f47818" containerID="a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680" exitCode=0
Oct 30 01:03:41.530342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.530176    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"
Oct 30 01:03:41.530342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.530202    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t" event={"ID":"35da3089-0e0e-468c-bed3-91f552f47818","Type":"ContainerDied","Data":"a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680"}
Oct 30 01:03:41.530342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.530287    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t" event={"ID":"35da3089-0e0e-468c-bed3-91f552f47818","Type":"ContainerDied","Data":"7eecf5b4862ee19d09010d06791c4db44041600e4b6814df3ddc4b906b7ec271"}
Oct 30 01:03:41.530342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.530317    4576 scope.go:122] "RemoveContainer" containerID="a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680"
Oct 30 01:03:41.543897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.543868    4576 scope.go:122] "RemoveContainer" containerID="a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680"
Oct 30 01:03:41.544364 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:03:41.544333    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680\": container with ID starting with a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680 not found: ID does not exist" containerID="a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680"
Oct 30 01:03:41.544513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.544376    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680"} err="failed to get container status \"a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680\": rpc error: code = NotFound desc = could not find container \"a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680\": container with ID starting with a74ac60ecd73ea6c19de7d8ae4259c5db34a803d8c4b0a1cd78fbd287e5a6680 not found: ID does not exist"
Oct 30 01:03:41.587936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.587904    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"]
Oct 30 01:03:41.590829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:41.590772    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b814fb-54ccf7b8f6-mx48t"]
Oct 30 01:03:42.109828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:42.109792    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35da3089-0e0e-468c-bed3-91f552f47818" path="/var/lib/kubelet/pods/35da3089-0e0e-468c-bed3-91f552f47818/volumes"
Oct 30 01:03:42.868636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:42.868604    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:43.645108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:43.645072    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:43.799050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:43.799013    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:44.021705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:44.021653    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:44.140873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:44.140837    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:44.313938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:44.313838    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:45.790759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:45.790726    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:46.318691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:46.318654    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:46.434651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:46.434615    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:47.782885 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:47.782846    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:48.387686 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:03:48.387653    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:03:48.847627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.847592    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"]
Oct 30 01:03:48.848202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848186    4576 cpu_manager.go:414] "containerMap: removing container" podUID="35da3089-0e0e-468c-bed3-91f552f47818" containerName="payload-processing"
Oct 30 01:03:48.848270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848204    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35da3089-0e0e-468c-bed3-91f552f47818" containerName="payload-processing"
Oct 30 01:03:48.848270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848251    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy"
Oct 30 01:03:48.848270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848257    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy"
Oct 30 01:03:48.848270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848265    4576 cpu_manager.go:414] "containerMap: removing container" podUID="cf7016a3-130b-497a-8cfd-d35b56208fe0" containerName="payload-processing"
Oct 30 01:03:48.848270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848271    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7016a3-130b-497a-8cfd-d35b56208fe0" containerName="payload-processing"
Oct 30 01:03:48.848427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848360    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="35da3089-0e0e-468c-bed3-91f552f47818" containerName="payload-processing"
Oct 30 01:03:48.848427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848378    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d71d034-e6e6-4133-be6f-a22b75bd12d9" containerName="istio-proxy"
Oct 30 01:03:48.848427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.848386    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7016a3-130b-497a-8cfd-d35b56208fe0" containerName="payload-processing"
Oct 30 01:03:48.853543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.853521    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.887869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.887837    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-869c367e-openshift-default-dockercfg-b9qvn\""
Oct 30 01:03:48.893195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.893168    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"]
Oct 30 01:03:48.958165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958132    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-podinfo\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.958361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958182    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-token\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.958361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958333    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-data\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.958438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958380    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istiod-ca-cert\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.958485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958473    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-certs\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.958524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958507    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-credential-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.958592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958577    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-envoy\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.958627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958608    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfb69\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-kube-api-access-gfb69\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:48.958660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:48.958643    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060297    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-certs\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060340    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-credential-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060362    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-envoy\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060386    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfb69\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-kube-api-access-gfb69\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060436    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060600    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-podinfo\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060654    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-token\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060727    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-data\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060753    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istiod-ca-cert\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060767    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-credential-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.060841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060751    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-certs\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.061068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.060996    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-data\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.061274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.061250    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-socket\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.061418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.061402    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istiod-ca-cert\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.062893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.062871    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-envoy\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.063144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.063126    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-podinfo\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.080937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.080907    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-token\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.081111 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.081089    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfb69\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-kube-api-access-gfb69\") pod \"e2e-mt-869c367e-openshift-default-78c446d585-6sjqs\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") " pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.166950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.166873    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:49.347754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.347709    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:49.485727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.485699    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"]
Oct 30 01:03:49.486736 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:03:49.486712    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cd76d8_fd25_4ff6_8dc6_fb084b7ab2a9.slice/crio-f78244b9996c0a033a9f3c6aea366b3c71241396db50ededb8abb82875019892 WatchSource:0}: Error finding container f78244b9996c0a033a9f3c6aea366b3c71241396db50ededb8abb82875019892: Status 404 returned error can't find the container with id f78244b9996c0a033a9f3c6aea366b3c71241396db50ededb8abb82875019892
Oct 30 01:03:49.489752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.489721    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:03:49.489840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.489790    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:03:49.489840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.489820    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:03:49.588274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.588218    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" event={"ID":"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9","Type":"ContainerStarted","Data":"04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf"}
Oct 30 01:03:49.588455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.588303    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" event={"ID":"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9","Type":"ContainerStarted","Data":"f78244b9996c0a033a9f3c6aea366b3c71241396db50ededb8abb82875019892"}
Oct 30 01:03:49.638388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:49.638325    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" podStartSLOduration=1.6383039689999999 podStartE2EDuration="1.638303969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:49.633386591 +0000 UTC m=+1509.704710451" watchObservedRunningTime="2026-10-30 01:03:49.638303969 +0000 UTC m=+1509.709627805"
Oct 30 01:03:50.163623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:50.163581    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:50.167226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:50.167195    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:50.193914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:50.193872    4576 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:03:50.194095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:50.193947    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 01:03:50.289257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:50.289185    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:50.523329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:50.522797    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:51.163513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:51.163481    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:51.183258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:51.179363    4576 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:03:51.192371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:51.192326    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 01:03:51.303260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:51.293557    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:51.406300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:51.406224    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:51.504740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:51.504701    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:03:52.167530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:52.167422    4576 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:03:52.167530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:52.167479    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 01:03:52.262976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:52.262938    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:52.680858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:52.680821    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:52.848495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:52.848462    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:53.185750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:53.185715    4576 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:03:53.186028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:53.186002    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 01:03:53.467922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:53.467882    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:53.634320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:53.634283    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:54.027707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:54.027678    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:54.179416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:54.179382    4576 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:03:54.179688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:54.179662    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 01:03:54.319326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:54.318666    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:55.171722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:55.171679    4576 patch_prober.go:29] interesting pod/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 01:03:55.172350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:55.172318    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 01:03:56.097316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:56.097270    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:56.341647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:56.341598    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:56.342114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:56.342019    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:56.343072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:56.343043    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:03:57.117187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:57.117151    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:57.438451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:57.438422    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:57.565740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:57.565705    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:57.946803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:57.946763    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:59.336475 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:59.336433    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:59.455689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:59.455657    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:03:59.835526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:03:59.835488    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:00.715795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:00.715761    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:00.830645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:00.830614    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:00.988475 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:00.988387    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"]
Oct 30 01:04:00.989154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:00.989123    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy" containerID="cri-o://5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4" gracePeriod=30
Oct 30 01:04:01.041652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:01.041622    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:01.785218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:01.785184    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:03.793623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:03.793587    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:04.211868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:04.211830    4576 patch_prober.go:29] interesting pod/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:04:04.212041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:04.211882    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:04:05.304113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:05.304080    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:05.596851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:05.596765    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:06.303916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.303888    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:04:06.418510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418473    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-token\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-token\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.418986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418549    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-kube-api-access-2jc5b\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-kube-api-access-2jc5b\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.418986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418589    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-socket\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.418986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418641    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-data\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.418986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418681    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-certs\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.418986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418788    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-credential-socket\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.418986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418843    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-envoy\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.418986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418932    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-socket" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:06.418986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.418951    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-podinfo\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.419618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.419050    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-certs" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:06.419618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.419130    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff643c71-d2c0-42b4-897e-ca9d3b180362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff643c71-d2c0-42b4-897e-ca9d3b180362-istiod-ca-cert\") pod \"ff643c71-d2c0-42b4-897e-ca9d3b180362\" (UID: \"ff643c71-d2c0-42b4-897e-ca9d3b180362\") "
Oct 30 01:04:06.419618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.419298    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-data" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:06.419618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.419363    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-credential-socket" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:06.420066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.420040    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.420280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.420265    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.420401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.420388    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.420486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.420474    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.420726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.420703    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff643c71-d2c0-42b4-897e-ca9d3b180362-istiod-ca-cert" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:06.421771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.421746    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-envoy" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:06.421862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.421802    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-podinfo" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:04:06.422101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.422084    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-kube-api-access-2jc5b" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "kube-api-access-2jc5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:06.422660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.422646    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-token" pod "ff643c71-d2c0-42b4-897e-ca9d3b180362" (UID: "ff643c71-d2c0-42b4-897e-ca9d3b180362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:06.522311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.522278    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.522311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.522308    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jc5b\" (UniqueName: \"kubernetes.io/projected/ff643c71-d2c0-42b4-897e-ca9d3b180362-kube-api-access-2jc5b\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.522311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.522318    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.522568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.522326    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff643c71-d2c0-42b4-897e-ca9d3b180362-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.522568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.522336    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff643c71-d2c0-42b4-897e-ca9d3b180362-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.523854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.523788    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:06.903108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.903023    4576 generic.go:356] "Generic (PLEG): container finished" podID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerID="5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4" exitCode=0
Oct 30 01:04:06.903108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.903085    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"
Oct 30 01:04:06.903355 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.903156    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" event={"ID":"ff643c71-d2c0-42b4-897e-ca9d3b180362","Type":"ContainerDied","Data":"5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4"}
Oct 30 01:04:06.903355 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.903258    4576 scope.go:122] "RemoveContainer" containerID="5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4"
Oct 30 01:04:06.903355 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.903223    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck" event={"ID":"ff643c71-d2c0-42b4-897e-ca9d3b180362","Type":"ContainerDied","Data":"fd7548bb983c105fe5e48e9bfdc69d78b52eace93e4e2bafb5d1aaf27a186d34"}
Oct 30 01:04:06.917068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.917045    4576 scope.go:122] "RemoveContainer" containerID="18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d"
Oct 30 01:04:06.943989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.943965    4576 scope.go:122] "RemoveContainer" containerID="5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4"
Oct 30 01:04:06.944643 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:04:06.944514    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4\": container with ID starting with 5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4 not found: ID does not exist" containerID="5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4"
Oct 30 01:04:06.944643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.944555    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4"} err="failed to get container status \"5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4\": rpc error: code = NotFound desc = could not find container \"5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4\": container with ID starting with 5cbd38018702ea2750a147f2bf733a10ad6477de0871c1fde9c94e6a682670f4 not found: ID does not exist"
Oct 30 01:04:06.944643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.944579    4576 scope.go:122] "RemoveContainer" containerID="18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d"
Oct 30 01:04:06.945288 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:04:06.945200    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d\": container with ID starting with 18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d not found: ID does not exist" containerID="18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d"
Oct 30 01:04:06.945288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.945262    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d"} err="failed to get container status \"18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d\": rpc error: code = NotFound desc = could not find container \"18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d\": container with ID starting with 18ca6e46147e9a14132ded079188589706772fcf047dc23e5bf75ac5d6efdd5d not found: ID does not exist"
Oct 30 01:04:06.982271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.981893    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"]
Oct 30 01:04:06.992746 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:06.992715    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7452d0-openshift-default-78649bfcd5-nmlck"]
Oct 30 01:04:07.010100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:07.008156    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:07.443352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:07.443316    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:07.558555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:07.558517    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:08.110947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:08.110893    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" path="/var/lib/kubelet/pods/ff643c71-d2c0-42b4-897e-ca9d3b180362/volumes"
Oct 30 01:04:09.404875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:09.404839    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:11.086815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:11.086770    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:11.201820 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:11.201785    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:11.321283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:11.321218    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:12.308526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:12.308481    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:12.441514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:12.441467    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:13.778984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:13.778932    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:13.981109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:13.981076    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:14.093652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:14.093551    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:15.458559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:15.458519    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:16.384717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:16.384673    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:16.500406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:16.500375    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:17.107613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:17.107580    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:18.383695 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:04:18.383656    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:04:18.761429 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:18.761381    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:18.917926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:18.917890    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:20.599992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:20.599956    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:20.877016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:20.876913    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:21.285697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:21.285658    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:21.749797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:21.749756    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:21.907258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:21.888108    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:22.366295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:22.366263    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:23.419897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:23.419864    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:23.536640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:23.536598    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:23.684014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:23.683932    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"]
Oct 30 01:04:23.684455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:23.684428    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy" containerID="cri-o://89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689" gracePeriod=30
Oct 30 01:04:23.746297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:23.746261    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:24.126485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:24.126452    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:25.995406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:25.995371    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:26.161882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:26.161830    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:26.283042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:26.282955    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:26.541523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:26.541438    4576 patch_prober.go:29] interesting pod/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:04:26.541523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:26.541507    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:04:27.886007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:27.885974    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:28.017960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:28.017925    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:28.167388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:28.167305    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:28.994339 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:28.994316    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:04:29.090602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.090573    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-certs\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.090797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.090679    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-credential-socket\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.090797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.090748    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-envoy\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.090921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.090812    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/639ecc18-2109-4983-905b-6d7ae273e375-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639ecc18-2109-4983-905b-6d7ae273e375-istio-podinfo\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.090921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.090860    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-kube-api-access-mddvx\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-kube-api-access-mddvx\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.090921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.090890    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-data\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.090921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.090900    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-certs" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:29.091439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.090972    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-socket\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.091587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.091463    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639ecc18-2109-4983-905b-6d7ae273e375-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639ecc18-2109-4983-905b-6d7ae273e375-istiod-ca-cert\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.091587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.091519    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-istio-token\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-istio-token\") pod \"639ecc18-2109-4983-905b-6d7ae273e375\" (UID: \"639ecc18-2109-4983-905b-6d7ae273e375\") "
Oct 30 01:04:29.091767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.091018    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-credential-socket" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:29.091874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.091220    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-socket" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:29.091952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.091252    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-data" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:29.092517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.092199    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639ecc18-2109-4983-905b-6d7ae273e375-istiod-ca-cert" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:29.092517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.092436    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.092517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.092457    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.092517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.092471    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639ecc18-2109-4983-905b-6d7ae273e375-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.092517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.092484    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.092517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.092495    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.093853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.093826    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/639ecc18-2109-4983-905b-6d7ae273e375-istio-podinfo" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:04:29.094225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.094103    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-envoy" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:29.094225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.094138    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-kube-api-access-mddvx" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "kube-api-access-mddvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:29.094421 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.094388    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-istio-token" pod "639ecc18-2109-4983-905b-6d7ae273e375" (UID: "639ecc18-2109-4983-905b-6d7ae273e375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:29.101541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.101518    4576 generic.go:356] "Generic (PLEG): container finished" podID="639ecc18-2109-4983-905b-6d7ae273e375" containerID="89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689" exitCode=0
Oct 30 01:04:29.101650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.101568    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"
Oct 30 01:04:29.101650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.101598    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" event={"ID":"639ecc18-2109-4983-905b-6d7ae273e375","Type":"ContainerDied","Data":"89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689"}
Oct 30 01:04:29.101729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.101655    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn" event={"ID":"639ecc18-2109-4983-905b-6d7ae273e375","Type":"ContainerDied","Data":"7205f1b2c19e8ea68c6bebb2b6e11c4d3646ec12b7e03f50b61c7657eec5f7b8"}
Oct 30 01:04:29.101729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.101686    4576 scope.go:122] "RemoveContainer" containerID="89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689"
Oct 30 01:04:29.127626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.127568    4576 scope.go:122] "RemoveContainer" containerID="89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689"
Oct 30 01:04:29.127924 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:04:29.127903    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689\": container with ID starting with 89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689 not found: ID does not exist" containerID="89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689"
Oct 30 01:04:29.127996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.127930    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689"} err="failed to get container status \"89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689\": rpc error: code = NotFound desc = could not find container \"89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689\": container with ID starting with 89f9c3a0776db4da396a667c30fe8a94e9f2eb1bf9fd35ad0cec0501f0aac689 not found: ID does not exist"
Oct 30 01:04:29.164828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.164805    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"]
Oct 30 01:04:29.170466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.170445    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b814fb-openshift-default-5775bb6787-x25gn"]
Oct 30 01:04:29.193624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.193595    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639ecc18-2109-4983-905b-6d7ae273e375-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.193624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.193619    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639ecc18-2109-4983-905b-6d7ae273e375-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.193799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.193634    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mddvx\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-kube-api-access-mddvx\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.193799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.193646    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639ecc18-2109-4983-905b-6d7ae273e375-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.784180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.784147    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:29.891768 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:29.891726    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:30.109678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:30.109596    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639ecc18-2109-4983-905b-6d7ae273e375" path="/var/lib/kubelet/pods/639ecc18-2109-4983-905b-6d7ae273e375/volumes"
Oct 30 01:04:30.168402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:30.168361    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:31.893514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:31.893469    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:32.645973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:32.645939    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:04:33.084360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:33.084326    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:33.579023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:33.578989    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:34.631011 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:34.630980    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:34.894324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:34.894220    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:35.003512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:35.003478    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:35.192325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:35.192291    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:35.306373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:35.306342    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:36.861502 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:36.861462    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:38.572259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:38.572216    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:40.418391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:40.418363    4576 scope.go:122] "RemoveContainer" containerID="bfb067d09dd5d53bb31e7ba5979399eeff4ffe0c363b513c869aac2efc3f7ffd"
Oct 30 01:04:40.458501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:40.458469    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:42.174820 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:42.174789    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:42.287798 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:42.287764    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:44.138130 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:44.138097    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:45.904482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:45.904445    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:47.308674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:47.308640    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:47.447257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:47.443978    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:47.704489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:47.704447    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:48.381495 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:04:48.381460    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:04:49.297074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:49.297045    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:50.817413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:50.817365    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:52.530868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:52.530771    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:53.952846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:53.952807    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:54.716030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:54.716004    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:54.833099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:54.833065    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:54.956596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:54.956554    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:55.312490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:55.312461    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:57.096801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:57.096765    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:04:58.903794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:04:58.903754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:00.249330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:00.249300    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"]
Oct 30 01:05:00.346480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:00.346442    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:00.636785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:00.636704    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:02.394777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:02.394737    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:03.807127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:03.807092    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:04.168122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:04.168031    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:04.486082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:04.486044    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:04.956318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:04.956281    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:05.130302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:05.130268    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:05.856327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:05.856291    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:06.472720 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:06.472683    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:06.587698 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:06.587639    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:06.711291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:06.711256    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:06.830485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:06.830401    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:07.123454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:07.123376    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:08.459656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:08.459628    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:09.036551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:09.036503    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:09.154767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:09.154728    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:09.403841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:09.402956    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:05:10.565980 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:10.565929    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:11.327183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:11.327153    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:11.447046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:11.447008    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:12.541271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:12.541219    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:13.881843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:13.881803    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:14.017708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:14.017666    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:14.181934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:14.181854    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:15.023058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.023011    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:15.131480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.131435    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:15.282346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.282211    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99" podUID="3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" containerName="main" containerID="cri-o://bef23a8269531570a7478955c3ae1ec6fc9352c0fd5e5a28ed1f01ad1918ed51" gracePeriod=45
Oct 30 01:05:15.453218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.453185    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:15.468834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.468798    4576 generic.go:356] "Generic (PLEG): container finished" podID="3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" containerID="bef23a8269531570a7478955c3ae1ec6fc9352c0fd5e5a28ed1f01ad1918ed51" exitCode=0
Oct 30 01:05:15.469123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.469098    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99" event={"ID":"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475","Type":"ContainerDied","Data":"bef23a8269531570a7478955c3ae1ec6fc9352c0fd5e5a28ed1f01ad1918ed51"}
Oct 30 01:05:15.721259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.718826    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:15.807928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.807894    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:05:15.812546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.811399    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tmp-dir\") pod \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") "
Oct 30 01:05:15.812546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.811517    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-model-cache\") pod \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") "
Oct 30 01:05:15.812546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.811584    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-kube-api-access-p2hhz\" (UniqueName: \"kubernetes.io/projected/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-kube-api-access-p2hhz\") pod \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") "
Oct 30 01:05:15.812546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.811640    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tls-certs\" (UniqueName: \"kubernetes.io/secret/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tls-certs\") pod \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") "
Oct 30 01:05:15.812546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.811669    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-home\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-home\") pod \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") "
Oct 30 01:05:15.812546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.811699    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-model-cache" pod "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" (UID: "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:15.812546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.811717    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-dshm\") pod \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\" (UID: \"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475\") "
Oct 30 01:05:15.812546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.812481    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tmp-dir" pod "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" (UID: "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:15.813185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.812614    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-home" pod "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" (UID: "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:15.814028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.813882    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:15.814359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.814323    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-kube-api-access-p2hhz" pod "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" (UID: "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475"). InnerVolumeSpecName "kube-api-access-p2hhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:15.814644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.814621    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-dshm" pod "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" (UID: "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:15.814868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.814792    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tls-certs" pod "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" (UID: "3246b13c-9f6f-4729-bcf2-4cbf8bd1e475"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:15.922538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.922497    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:15.922538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.922534    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:15.922787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.922550    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2hhz\" (UniqueName: \"kubernetes.io/projected/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-kube-api-access-p2hhz\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:15.922787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.922567    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:15.922787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:15.922580    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:16.481028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:16.480998    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"
Oct 30 01:05:16.481597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:16.481026    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99" event={"ID":"3246b13c-9f6f-4729-bcf2-4cbf8bd1e475","Type":"ContainerDied","Data":"a4f8afe57125d815b8e7950caf0b87a61e1bf0c945a846e39a84174722dd2ca9"}
Oct 30 01:05:16.481597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:16.481086    4576 scope.go:122] "RemoveContainer" containerID="bef23a8269531570a7478955c3ae1ec6fc9352c0fd5e5a28ed1f01ad1918ed51"
Oct 30 01:05:16.538644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:16.538612    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"]
Oct 30 01:05:16.552714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:16.552684    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-869c367e/e2e-lifecycle-model-869c367e-kserve-7b66846949-f6w99"]
Oct 30 01:05:17.064545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:17.064505    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:17.317313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:17.314223    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:17.730750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:17.730718    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:18.139994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:18.127858    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" path="/var/lib/kubelet/pods/3246b13c-9f6f-4729-bcf2-4cbf8bd1e475/volumes"
Oct 30 01:05:18.201884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:18.201836    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:18.325855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:18.325822    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:18.563569 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:05:18.563524    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:05:18.901169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:18.901069    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:19.476763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:19.476724    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:19.632287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:19.630723    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:20.743539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:20.743506    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:21.179297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:21.179186    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:21.298749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:21.298720    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:22.658598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:22.658553    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:22.788253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:22.788196    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:22.928541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:22.927804    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:24.378463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:24.378423    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:24.576109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:24.576075    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:24.698914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:24.698882    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:26.173877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:26.173838    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:27.502504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:27.502469    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"]
Oct 30 01:05:27.503390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:27.503358    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl" podUID="8a073ffb-6cf4-485d-9ca5-55ec6006e25c" containerName="maas-api" containerID="cri-o://1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47" gracePeriod=30
Oct 30 01:05:27.578413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:27.578371    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:28.025113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.025088    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:05:28.159924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.159879    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-kube-api-access-whs26\" (UniqueName: \"kubernetes.io/projected/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-kube-api-access-whs26\") pod \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") "
Oct 30 01:05:28.160121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.159956    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-openshift-service-ca\") pod \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") "
Oct 30 01:05:28.160121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.160067    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-maas-api-tls\") pod \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") "
Oct 30 01:05:28.160261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.160225    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-metrics-certs\") pod \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\" (UID: \"8a073ffb-6cf4-485d-9ca5-55ec6006e25c\") "
Oct 30 01:05:28.162563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.162533    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-openshift-service-ca" pod "8a073ffb-6cf4-485d-9ca5-55ec6006e25c" (UID: "8a073ffb-6cf4-485d-9ca5-55ec6006e25c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:28.169528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.169475    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-maas-api-tls" pod "8a073ffb-6cf4-485d-9ca5-55ec6006e25c" (UID: "8a073ffb-6cf4-485d-9ca5-55ec6006e25c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:28.169687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.169567    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-metrics-certs" pod "8a073ffb-6cf4-485d-9ca5-55ec6006e25c" (UID: "8a073ffb-6cf4-485d-9ca5-55ec6006e25c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:28.170639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.170601    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-kube-api-access-whs26" pod "8a073ffb-6cf4-485d-9ca5-55ec6006e25c" (UID: "8a073ffb-6cf4-485d-9ca5-55ec6006e25c"). InnerVolumeSpecName "kube-api-access-whs26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:28.263147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.263050    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-maas-api-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:28.263147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.263084    4576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-metrics-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:28.263147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.263099    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whs26\" (UniqueName: \"kubernetes.io/projected/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-kube-api-access-whs26\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:28.263147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.263112    4576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a073ffb-6cf4-485d-9ca5-55ec6006e25c-openshift-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:28.581063 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.580973    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:28.638702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.638664    4576 generic.go:356] "Generic (PLEG): container finished" podID="8a073ffb-6cf4-485d-9ca5-55ec6006e25c" containerID="1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47" exitCode=0
Oct 30 01:05:28.638903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.638707    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl" event={"ID":"8a073ffb-6cf4-485d-9ca5-55ec6006e25c","Type":"ContainerDied","Data":"1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47"}
Oct 30 01:05:28.638903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.638737    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"
Oct 30 01:05:28.638903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.638804    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl" event={"ID":"8a073ffb-6cf4-485d-9ca5-55ec6006e25c","Type":"ContainerDied","Data":"c43ae1560751c0ab798dc6f87c7a4169036ed06b8c7c7dbe60eaf593ba302ff4"}
Oct 30 01:05:28.638903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.638857    4576 scope.go:122] "RemoveContainer" containerID="1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47"
Oct 30 01:05:28.650280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.650260    4576 scope.go:122] "RemoveContainer" containerID="1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47"
Oct 30 01:05:28.650579 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:05:28.650561    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47\": container with ID starting with 1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47 not found: ID does not exist" containerID="1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47"
Oct 30 01:05:28.650645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.650593    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47"} err="failed to get container status \"1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47\": rpc error: code = NotFound desc = could not find container \"1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47\": container with ID starting with 1c7ccd8bdbce2147adc085048dfcf6bbb21184992eeaf1eb83d4f5108270ec47 not found: ID does not exist"
Oct 30 01:05:28.695758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.695728    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"]
Oct 30 01:05:28.709126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.709098    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ee0869-7f696b6dd-t5gzl"]
Oct 30 01:05:28.789960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:28.789927    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:29.023389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:29.023354    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:29.315687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:29.315606    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:30.108761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:30.108728    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a073ffb-6cf4-485d-9ca5-55ec6006e25c" path="/var/lib/kubelet/pods/8a073ffb-6cf4-485d-9ca5-55ec6006e25c/volumes"
Oct 30 01:05:30.915653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:30.915617    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:32.005631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:32.005598    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:32.567322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:32.567285    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"]
Oct 30 01:05:32.567520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:32.567492    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy" containerID="cri-o://8cce920921efc1f479819442eae58de0d30e5586adc50579b88a510468f31560" gracePeriod=30
Oct 30 01:05:32.601894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:32.601843    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:34.243436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:34.243398    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:34.393431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:34.393400    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:34.709541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:34.709493    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:35.613378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:35.613346    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:35.745478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:35.745444    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:35.904505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:35.904420    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:36.032257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:36.031852    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:36.438380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:36.438349    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:36.684098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:36.684061    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:36.811048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:36.810948    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:37.206568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:37.206522    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:37.320892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:37.320851    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:37.724277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:37.723977    4576 generic.go:356] "Generic (PLEG): container finished" podID="015cb6bc-6824-416b-a693-38f261dc435d" containerID="8cce920921efc1f479819442eae58de0d30e5586adc50579b88a510468f31560" exitCode=0
Oct 30 01:05:37.724277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:37.724161    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" event={"ID":"015cb6bc-6824-416b-a693-38f261dc435d","Type":"ContainerDied","Data":"8cce920921efc1f479819442eae58de0d30e5586adc50579b88a510468f31560"}
Oct 30 01:05:37.752658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:37.752573    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:37.965556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:37.965531    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 01:05:38.089357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089324    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-credential-socket\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.089555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089432    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-socket\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.089555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089472    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015cb6bc-6824-416b-a693-38f261dc435d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015cb6bc-6824-416b-a693-38f261dc435d-istiod-ca-cert\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.089555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089533    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-certs\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.089695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089603    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-envoy\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.089695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089655    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-istio-token\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-istio-token\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.089695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089676    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-credential-socket" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:38.089864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089710    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-kube-api-access-nnkrv\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-kube-api-access-nnkrv\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.089927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089855    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-certs" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:38.089927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.089914    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/015cb6bc-6824-416b-a693-38f261dc435d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015cb6bc-6824-416b-a693-38f261dc435d-istio-podinfo\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.090043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.090025    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-data\") pod \"015cb6bc-6824-416b-a693-38f261dc435d\" (UID: \"015cb6bc-6824-416b-a693-38f261dc435d\") "
Oct 30 01:05:38.090620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.090587    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015cb6bc-6824-416b-a693-38f261dc435d-istiod-ca-cert" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:38.090804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.090638    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-socket" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:38.090804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.090668    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-data" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:38.090804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.090687    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.090804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.090733    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.093643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.093603    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-envoy" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:38.094559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.094533    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-istio-token" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:38.094904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.094884    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-kube-api-access-nnkrv" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "kube-api-access-nnkrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:38.095661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.095624    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/015cb6bc-6824-416b-a693-38f261dc435d-istio-podinfo" pod "015cb6bc-6824-416b-a693-38f261dc435d" (UID: "015cb6bc-6824-416b-a693-38f261dc435d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:38.128467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.128438    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:38.192319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.192286    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.192319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.192320    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015cb6bc-6824-416b-a693-38f261dc435d-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.192609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.192333    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.192609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.192344    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.192609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.192356    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnkrv\" (UniqueName: \"kubernetes.io/projected/015cb6bc-6824-416b-a693-38f261dc435d-kube-api-access-nnkrv\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.192609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.192370    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015cb6bc-6824-416b-a693-38f261dc435d-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.192609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.192384    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015cb6bc-6824-416b-a693-38f261dc435d-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:38.732802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.732771    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"
Oct 30 01:05:38.733321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.732807    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w" event={"ID":"015cb6bc-6824-416b-a693-38f261dc435d","Type":"ContainerDied","Data":"8b90398685e0eab28091f7fb795f4a39a14a3ad1dda031f5196810bd2d5c77fe"}
Oct 30 01:05:38.733321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.732873    4576 scope.go:122] "RemoveContainer" containerID="8cce920921efc1f479819442eae58de0d30e5586adc50579b88a510468f31560"
Oct 30 01:05:38.804518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.804485    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"]
Oct 30 01:05:38.809134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:38.809103    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-6ps6w"]
Oct 30 01:05:39.842218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:39.842184    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:39.956692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:39.956658    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:40.109179 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:40.109102    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015cb6bc-6824-416b-a693-38f261dc435d" path="/var/lib/kubelet/pods/015cb6bc-6824-416b-a693-38f261dc435d/volumes"
Oct 30 01:05:41.259059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:41.259019    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:41.445157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:41.445124    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:41.648855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:41.648754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:42.150551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:42.150513    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:42.298593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:42.298550    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:43.668302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:43.668267    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:43.809519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:43.809480    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:44.652937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:44.652894    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:44.767575 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:44.767539    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:44.896833 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:44.896708    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:44.924685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:44.924605    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"]
Oct 30 01:05:44.925353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:44.925067    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy" containerID="cri-o://04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf" gracePeriod=30
Oct 30 01:05:45.179948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:45.179828    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:45.269646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:45.269614    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:05:46.317639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.317605    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:46.480155 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.480116    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:46.506450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.506400    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"]
Oct 30 01:05:46.506995 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.506973    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy"
Oct 30 01:05:46.506995 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.506992    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507004    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8a073ffb-6cf4-485d-9ca5-55ec6006e25c" containerName="maas-api"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507009    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a073ffb-6cf4-485d-9ca5-55ec6006e25c" containerName="maas-api"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507034    4576 cpu_manager.go:414] "containerMap: removing container" podUID="3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" containerName="main"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507043    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" containerName="main"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507053    4576 cpu_manager.go:414] "containerMap: removing container" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507058    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507070    4576 cpu_manager.go:414] "containerMap: removing container" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507074    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507083    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy"
Oct 30 01:05:46.507153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507087    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy"
Oct 30 01:05:46.507515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507176    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy"
Oct 30 01:05:46.507515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507184    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a073ffb-6cf4-485d-9ca5-55ec6006e25c" containerName="maas-api"
Oct 30 01:05:46.507515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507192    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="015cb6bc-6824-416b-a693-38f261dc435d" containerName="istio-proxy"
Oct 30 01:05:46.507515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507200    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="3246b13c-9f6f-4729-bcf2-4cbf8bd1e475" containerName="main"
Oct 30 01:05:46.507515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.507208    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="639ecc18-2109-4983-905b-6d7ae273e375" containerName="istio-proxy"
Oct 30 01:05:46.512364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.512339    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.530129 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.530098    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"]
Oct 30 01:05:46.540385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.540351    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:05:46.540538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.540432    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:05:46.641194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.641090    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hx8\" (UniqueName: \"kubernetes.io/projected/3b286824-c144-4779-ba31-7337eb04df94-kube-api-access-k4hx8\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.641405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.641189    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b286824-c144-4779-ba31-7337eb04df94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.641405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.641331    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b286824-c144-4779-ba31-7337eb04df94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.742904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.742863    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hx8\" (UniqueName: \"kubernetes.io/projected/3b286824-c144-4779-ba31-7337eb04df94-kube-api-access-k4hx8\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.743114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.742939    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b286824-c144-4779-ba31-7337eb04df94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.743114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.742999    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b286824-c144-4779-ba31-7337eb04df94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.743717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.743688    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b286824-c144-4779-ba31-7337eb04df94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.745878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.745849    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b286824-c144-4779-ba31-7337eb04df94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.768273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.766472    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hx8\" (UniqueName: \"kubernetes.io/projected/3b286824-c144-4779-ba31-7337eb04df94-kube-api-access-k4hx8\") pod \"maas-api-revoke-sub-dc6e6801-q5v7k\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:46.827360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:46.827323    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:47.145970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:47.145904    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"]
Oct 30 01:05:47.146643 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:05:47.146615    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b286824_c144_4779_ba31_7337eb04df94.slice/crio-2fe8d99541b18193e41d4b00017e46806127cdb14a460e12c9d4186852a2139b WatchSource:0}: Error finding container 2fe8d99541b18193e41d4b00017e46806127cdb14a460e12c9d4186852a2139b: Status 404 returned error can't find the container with id 2fe8d99541b18193e41d4b00017e46806127cdb14a460e12c9d4186852a2139b
Oct 30 01:05:47.296417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:47.296386    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:47.798541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:47.798502    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k" event={"ID":"3b286824-c144-4779-ba31-7337eb04df94","Type":"ContainerStarted","Data":"4870b5e048d9408b0bf04516b9a21ac188abe78479567905948e33335a8e8b3b"}
Oct 30 01:05:47.799040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:47.798562    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k" event={"ID":"3b286824-c144-4779-ba31-7337eb04df94","Type":"ContainerStarted","Data":"2fe8d99541b18193e41d4b00017e46806127cdb14a460e12c9d4186852a2139b"}
Oct 30 01:05:47.841295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:47.841193    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k" podStartSLOduration=1.841176426 podStartE2EDuration="1.841176426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:47.840350923 +0000 UTC m=+1627.911674763" watchObservedRunningTime="2026-10-30 01:05:47.841176426 +0000 UTC m=+1627.912500261"
Oct 30 01:05:48.115639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:48.115603    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:48.260279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:48.260248    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:48.442632 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:05:48.436315    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:05:48.805412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:48.805315    4576 generic.go:356] "Generic (PLEG): container finished" podID="3b286824-c144-4779-ba31-7337eb04df94" containerID="4870b5e048d9408b0bf04516b9a21ac188abe78479567905948e33335a8e8b3b" exitCode=0
Oct 30 01:05:48.805782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:48.805406    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k" event={"ID":"3b286824-c144-4779-ba31-7337eb04df94","Type":"ContainerDied","Data":"4870b5e048d9408b0bf04516b9a21ac188abe78479567905948e33335a8e8b3b"}
Oct 30 01:05:49.374576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:49.374545    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:49.505281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:49.505248    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:49.888573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:49.888538    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:50.044261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.039294    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:50.092633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.092606    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:50.221272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.214133    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b286824-c144-4779-ba31-7337eb04df94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b286824-c144-4779-ba31-7337eb04df94-maas-api-xks-ca\") pod \"3b286824-c144-4779-ba31-7337eb04df94\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") "
Oct 30 01:05:50.221272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.214187    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b286824-c144-4779-ba31-7337eb04df94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b286824-c144-4779-ba31-7337eb04df94-maas-api-openshift-ca\") pod \"3b286824-c144-4779-ba31-7337eb04df94\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") "
Oct 30 01:05:50.221272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.214309    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b286824-c144-4779-ba31-7337eb04df94-kube-api-access-k4hx8\" (UniqueName: \"kubernetes.io/projected/3b286824-c144-4779-ba31-7337eb04df94-kube-api-access-k4hx8\") pod \"3b286824-c144-4779-ba31-7337eb04df94\" (UID: \"3b286824-c144-4779-ba31-7337eb04df94\") "
Oct 30 01:05:50.221272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.214862    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b286824-c144-4779-ba31-7337eb04df94-maas-api-openshift-ca" pod "3b286824-c144-4779-ba31-7337eb04df94" (UID: "3b286824-c144-4779-ba31-7337eb04df94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:50.221272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.217535    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b286824-c144-4779-ba31-7337eb04df94-maas-api-xks-ca" pod "3b286824-c144-4779-ba31-7337eb04df94" (UID: "3b286824-c144-4779-ba31-7337eb04df94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:50.221272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.218572    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b286824-c144-4779-ba31-7337eb04df94-kube-api-access-k4hx8" pod "3b286824-c144-4779-ba31-7337eb04df94" (UID: "3b286824-c144-4779-ba31-7337eb04df94"). InnerVolumeSpecName "kube-api-access-k4hx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:50.255440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.255405    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:05:50.317352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.316585    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b286824-c144-4779-ba31-7337eb04df94-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.317352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.316621    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b286824-c144-4779-ba31-7337eb04df94-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.317352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.316640    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4hx8\" (UniqueName: \"kubernetes.io/projected/3b286824-c144-4779-ba31-7337eb04df94-kube-api-access-k4hx8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.424260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.423859    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-certs\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.424260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.423979    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istiod-ca-cert\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.424260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.424060    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-data\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.435261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.428270    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-socket\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.435261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.428412    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-kube-api-access-gfb69\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-kube-api-access-gfb69\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.435261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.428467    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-envoy\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.435261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.428550    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-credential-socket\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.435261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.428587    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-podinfo\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.439074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.438765    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-certs" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:50.439927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.439897    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istiod-ca-cert" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:50.448463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.448432    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-token\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-token\") pod \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\" (UID: \"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9\") "
Oct 30 01:05:50.450168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.449414    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.452646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.452628    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.452793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.451957    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-socket" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:50.453455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.453433    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-envoy" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:50.455359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.454388    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-data" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:50.456357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.456325    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-credential-socket" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:50.472427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.470288    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-podinfo" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:50.472427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.470544    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-kube-api-access-gfb69" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "kube-api-access-gfb69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:50.472664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.472613    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-token" pod "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" (UID: "76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:50.554245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.554173    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.554245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.554213    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.554245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.554250    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.554619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.554263    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.554619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.554276    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.554619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.554289    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfb69\" (UniqueName: \"kubernetes.io/projected/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-kube-api-access-gfb69\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.554619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.554303    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:05:50.820864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.820831    4576 generic.go:356] "Generic (PLEG): container finished" podID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerID="04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf" exitCode=0
Oct 30 01:05:50.821157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.820900    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"
Oct 30 01:05:50.821157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.820915    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" event={"ID":"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9","Type":"ContainerDied","Data":"04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf"}
Oct 30 01:05:50.821157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.820982    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs" event={"ID":"76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9","Type":"ContainerDied","Data":"f78244b9996c0a033a9f3c6aea366b3c71241396db50ededb8abb82875019892"}
Oct 30 01:05:50.821157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.821015    4576 scope.go:122] "RemoveContainer" containerID="04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf"
Oct 30 01:05:50.822845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.822770    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"
Oct 30 01:05:50.822845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.822777    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k" event={"ID":"3b286824-c144-4779-ba31-7337eb04df94","Type":"ContainerDied","Data":"2fe8d99541b18193e41d4b00017e46806127cdb14a460e12c9d4186852a2139b"}
Oct 30 01:05:50.822845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.822824    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe8d99541b18193e41d4b00017e46806127cdb14a460e12c9d4186852a2139b"
Oct 30 01:05:50.836439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.836410    4576 scope.go:122] "RemoveContainer" containerID="04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf"
Oct 30 01:05:50.836895 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:05:50.836876    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf\": container with ID starting with 04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf not found: ID does not exist" containerID="04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf"
Oct 30 01:05:50.836989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.836902    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf"} err="failed to get container status \"04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf\": rpc error: code = NotFound desc = could not find container \"04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf\": container with ID starting with 04ce8a6645411c8c584a940f5d7d9fadf60b49f780771c3c76c737219d1fbfbf not found: ID does not exist"
Oct 30 01:05:50.883537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.883506    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"]
Oct 30 01:05:50.894811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:50.894780    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-869c367e-openshift-default-78c446d585-6sjqs"]
Oct 30 01:05:51.377695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:51.377664    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:52.110050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:52.110015    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" path="/var/lib/kubelet/pods/76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9/volumes"
Oct 30 01:05:52.953891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:52.953854    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:53.085447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:53.085411    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:53.210912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:53.209690    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:54.455894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:54.455852    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:54.582023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:54.581977    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:55.022250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:55.022204    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:56.214280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.213956    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:56.331371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.331315    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:56.374141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.374101    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"]
Oct 30 01:05:56.374758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.374739    4576 cpu_manager.go:414] "containerMap: removing container" podUID="3b286824-c144-4779-ba31-7337eb04df94" containerName="revoke-keys"
Oct 30 01:05:56.374893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.374760    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b286824-c144-4779-ba31-7337eb04df94" containerName="revoke-keys"
Oct 30 01:05:56.374893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.374801    4576 cpu_manager.go:414] "containerMap: removing container" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy"
Oct 30 01:05:56.374893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.374809    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy"
Oct 30 01:05:56.375057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.374955    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff643c71-d2c0-42b4-897e-ca9d3b180362" containerName="istio-proxy"
Oct 30 01:05:56.375057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.374970    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b286824-c144-4779-ba31-7337eb04df94" containerName="revoke-keys"
Oct 30 01:05:56.375057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.374986    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cd76d8-fd25-4ff6-8dc6-fb084b7ab2a9" containerName="istio-proxy"
Oct 30 01:05:56.380028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.379999    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:56.396112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.396068    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"]
Oct 30 01:05:56.430714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.430683    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-286d36\""
Oct 30 01:05:56.479338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.456298    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zsp8\" (UniqueName: \"kubernetes.io/projected/222f8572-e0eb-4f90-a080-7d3d7ce1a047-kube-api-access-9zsp8\") pod \"payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv\" (UID: \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:56.479338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.456416    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222f8572-e0eb-4f90-a080-7d3d7ce1a047-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv\" (UID: \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:56.513493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.486578    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-286d36-dockercfg-tbc7l\""
Oct 30 01:05:56.571703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.563205    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zsp8\" (UniqueName: \"kubernetes.io/projected/222f8572-e0eb-4f90-a080-7d3d7ce1a047-kube-api-access-9zsp8\") pod \"payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv\" (UID: \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:56.571703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.563322    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222f8572-e0eb-4f90-a080-7d3d7ce1a047-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv\" (UID: \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:56.571703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.564323    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222f8572-e0eb-4f90-a080-7d3d7ce1a047-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv\" (UID: \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:56.589555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.589524    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zsp8\" (UniqueName: \"kubernetes.io/projected/222f8572-e0eb-4f90-a080-7d3d7ce1a047-kube-api-access-9zsp8\") pod \"payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv\" (UID: \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:56.700702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:56.700666    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:57.018558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:57.018527    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"]
Oct 30 01:05:57.019788 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:05:57.019761    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222f8572_e0eb_4f90_a080_7d3d7ce1a047.slice/crio-3eeb3c806dc318e986dee8685fa4f4002deb9554d6de577f710158599d34be61 WatchSource:0}: Error finding container 3eeb3c806dc318e986dee8685fa4f4002deb9554d6de577f710158599d34be61: Status 404 returned error can't find the container with id 3eeb3c806dc318e986dee8685fa4f4002deb9554d6de577f710158599d34be61
Oct 30 01:05:57.260275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:57.260227    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:57.372403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:57.372325    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:57.823692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:57.823656    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:57.879257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:57.878498    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv" event={"ID":"222f8572-e0eb-4f90-a080-7d3d7ce1a047","Type":"ContainerStarted","Data":"c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db"}
Oct 30 01:05:57.879257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:57.878559    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv" event={"ID":"222f8572-e0eb-4f90-a080-7d3d7ce1a047","Type":"ContainerStarted","Data":"3eeb3c806dc318e986dee8685fa4f4002deb9554d6de577f710158599d34be61"}
Oct 30 01:05:57.883310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:57.879522    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:05:57.953257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:57.951958    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv" podStartSLOduration=1.951939767 podStartE2EDuration="1.951939767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:57.934654271 +0000 UTC m=+1638.005978107" watchObservedRunningTime="2026-10-30 01:05:57.951939767 +0000 UTC m=+1638.023263604"
Oct 30 01:05:58.098266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:58.098163    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:59.067459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:59.067423    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:59.178561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:59.178528    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:05:59.311373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:05:59.311335    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:01.064516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:01.064477    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:01.576660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:01.576628    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:01.687708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:01.687671    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:02.926634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:02.926604    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:03.460316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.460276    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:03.554198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.554164    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"]
Oct 30 01:06:03.557917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.557899    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:03.577815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.577792    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"]
Oct 30 01:06:03.586819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.586795    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-42d56b\""
Oct 30 01:06:03.594482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.594456    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:03.624058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.624022    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"]
Oct 30 01:06:03.628316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.628290    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:03.646772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.646746    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"]
Oct 30 01:06:03.651134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.651100    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-42d56b-dockercfg-vflmd\""
Oct 30 01:06:03.693606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.693521    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641d53d4-0a3c-496d-898d-bbc381ba8102-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6\" (UID: \"641d53d4-0a3c-496d-898d-bbc381ba8102\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:03.693871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.693681    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpgkt\" (UniqueName: \"kubernetes.io/projected/641d53d4-0a3c-496d-898d-bbc381ba8102-kube-api-access-fpgkt\") pod \"payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6\" (UID: \"641d53d4-0a3c-496d-898d-bbc381ba8102\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:03.799836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.795176    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f55c7e-f518-4485-a436-fa6181614bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm\" (UID: \"57f55c7e-f518-4485-a436-fa6181614bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:03.799836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.795320    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641d53d4-0a3c-496d-898d-bbc381ba8102-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6\" (UID: \"641d53d4-0a3c-496d-898d-bbc381ba8102\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:03.799836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.795415    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7m6l\" (UniqueName: \"kubernetes.io/projected/57f55c7e-f518-4485-a436-fa6181614bdf-kube-api-access-l7m6l\") pod \"payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm\" (UID: \"57f55c7e-f518-4485-a436-fa6181614bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:03.799836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.795446    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgkt\" (UniqueName: \"kubernetes.io/projected/641d53d4-0a3c-496d-898d-bbc381ba8102-kube-api-access-fpgkt\") pod \"payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6\" (UID: \"641d53d4-0a3c-496d-898d-bbc381ba8102\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:03.799836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.796694    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641d53d4-0a3c-496d-898d-bbc381ba8102-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6\" (UID: \"641d53d4-0a3c-496d-898d-bbc381ba8102\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:03.818171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.818140    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgkt\" (UniqueName: \"kubernetes.io/projected/641d53d4-0a3c-496d-898d-bbc381ba8102-kube-api-access-fpgkt\") pod \"payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6\" (UID: \"641d53d4-0a3c-496d-898d-bbc381ba8102\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:03.872533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.872494    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:03.898388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.898354    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f55c7e-f518-4485-a436-fa6181614bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm\" (UID: \"57f55c7e-f518-4485-a436-fa6181614bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:03.898602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.898574    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7m6l\" (UniqueName: \"kubernetes.io/projected/57f55c7e-f518-4485-a436-fa6181614bdf-kube-api-access-l7m6l\") pod \"payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm\" (UID: \"57f55c7e-f518-4485-a436-fa6181614bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:03.901561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.901537    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f55c7e-f518-4485-a436-fa6181614bdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm\" (UID: \"57f55c7e-f518-4485-a436-fa6181614bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:03.926621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.926580    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7m6l\" (UniqueName: \"kubernetes.io/projected/57f55c7e-f518-4485-a436-fa6181614bdf-kube-api-access-l7m6l\") pod \"payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm\" (UID: \"57f55c7e-f518-4485-a436-fa6181614bdf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:03.942913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:03.942879    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:04.197411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.197377    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"]
Oct 30 01:06:04.198790 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:06:04.198761    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641d53d4_0a3c_496d_898d_bbc381ba8102.slice/crio-7eb96f6c44f5b45a0a571d4d1126f1e9f11f8938c3d34e2713236d9424f669da WatchSource:0}: Error finding container 7eb96f6c44f5b45a0a571d4d1126f1e9f11f8938c3d34e2713236d9424f669da: Status 404 returned error can't find the container with id 7eb96f6c44f5b45a0a571d4d1126f1e9f11f8938c3d34e2713236d9424f669da
Oct 30 01:06:04.268696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.268674    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"]
Oct 30 01:06:04.269138 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:06:04.269121    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f55c7e_f518_4485_a436_fa6181614bdf.slice/crio-3c60b51bf426f95956fb23997bc7a0d61f73b2174eb941e1c2cbc844515ed4ac WatchSource:0}: Error finding container 3c60b51bf426f95956fb23997bc7a0d61f73b2174eb941e1c2cbc844515ed4ac: Status 404 returned error can't find the container with id 3c60b51bf426f95956fb23997bc7a0d61f73b2174eb941e1c2cbc844515ed4ac
Oct 30 01:06:04.765504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.765465    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:04.939023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.938985    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6" event={"ID":"641d53d4-0a3c-496d-898d-bbc381ba8102","Type":"ContainerStarted","Data":"d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe"}
Oct 30 01:06:04.939540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.939155    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:04.939540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.939190    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6" event={"ID":"641d53d4-0a3c-496d-898d-bbc381ba8102","Type":"ContainerStarted","Data":"7eb96f6c44f5b45a0a571d4d1126f1e9f11f8938c3d34e2713236d9424f669da"}
Oct 30 01:06:04.940680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.940653    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm" event={"ID":"57f55c7e-f518-4485-a436-fa6181614bdf","Type":"ContainerStarted","Data":"c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38"}
Oct 30 01:06:04.940816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.940705    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:04.940816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.940721    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm" event={"ID":"57f55c7e-f518-4485-a436-fa6181614bdf","Type":"ContainerStarted","Data":"3c60b51bf426f95956fb23997bc7a0d61f73b2174eb941e1c2cbc844515ed4ac"}
Oct 30 01:06:04.982904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:04.982854    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6" podStartSLOduration=1.982838554 podStartE2EDuration="1.982838554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:04.97986371 +0000 UTC m=+1645.051187549" watchObservedRunningTime="2026-10-30 01:06:04.982838554 +0000 UTC m=+1645.054162394"
Oct 30 01:06:05.025031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:05.024918    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm" podStartSLOduration=2.024902937 podStartE2EDuration="2.024902937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:05.022020854 +0000 UTC m=+1645.093344702" watchObservedRunningTime="2026-10-30 01:06:05.024902937 +0000 UTC m=+1645.096226771"
Oct 30 01:06:06.356817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:06.356784    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:07.905004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:07.904974    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:09.559164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:09.559130    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:09.896871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:09.896796    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:06:11.130194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:11.130161    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:12.691889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:12.691849    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:14.577064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:14.577028    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:15.950549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:15.950518    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:06:15.951042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:15.950598    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:06:16.465448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:16.465411    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:16.590159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:16.590124    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:18.382819 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:06:18.382777    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:06:19.170360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:19.170325    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:21.451662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:21.451578    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:21.599225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:21.599182    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:22.086757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:22.086723    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:23.892361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:23.892310    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:25.461785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:25.461749    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:25.594049 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:25.594012    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:25.716612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:25.716520    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:27.578331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.578296    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"]
Oct 30 01:06:27.583425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.583406    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.598336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.598308    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"]
Oct 30 01:06:27.608586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.608554    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:06:27.608768 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.608607    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:06:27.696913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.696886    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:27.736649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.736605    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.736845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.736669    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.736845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.736819    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfmm\" (UniqueName: \"kubernetes.io/projected/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-kube-api-access-ksfmm\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.838130 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.838048    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.838130 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.838097    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.838403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.838136    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfmm\" (UniqueName: \"kubernetes.io/projected/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-kube-api-access-ksfmm\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.838836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.838816    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.840737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.840714    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.864203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.864165    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfmm\" (UniqueName: \"kubernetes.io/projected/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-kube-api-access-ksfmm\") pod \"maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:27.899566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:27.899532    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:28.208376 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:28.208344    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"]
Oct 30 01:06:28.210044 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:06:28.210010    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ce9d04_b4ba_48f3_b44f_3a37d0bfff00.slice/crio-e4d99b28c3d7ef6c259a1fc02f89de466deccbe61b4bba235bc0759e5625ad9b WatchSource:0}: Error finding container e4d99b28c3d7ef6c259a1fc02f89de466deccbe61b4bba235bc0759e5625ad9b: Status 404 returned error can't find the container with id e4d99b28c3d7ef6c259a1fc02f89de466deccbe61b4bba235bc0759e5625ad9b
Oct 30 01:06:29.146966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:29.146926    4576 generic.go:356] "Generic (PLEG): container finished" podID="77ce9d04-b4ba-48f3-b44f-3a37d0bfff00" containerID="f2ac475a0c9acc0bda59d1178bc3e28da75fee068d46ee9cef653a0d977b26d1" exitCode=0
Oct 30 01:06:29.147404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:29.147006    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx" event={"ID":"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00","Type":"ContainerDied","Data":"f2ac475a0c9acc0bda59d1178bc3e28da75fee068d46ee9cef653a0d977b26d1"}
Oct 30 01:06:29.147404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:29.147085    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx" event={"ID":"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00","Type":"ContainerStarted","Data":"e4d99b28c3d7ef6c259a1fc02f89de466deccbe61b4bba235bc0759e5625ad9b"}
Oct 30 01:06:29.606522 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:29.606487    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:30.325113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.325083    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:30.371705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.371678    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-openshift-ca\") pod \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") "
Oct 30 01:06:30.371873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.371783    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-kube-api-access-ksfmm\" (UniqueName: \"kubernetes.io/projected/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-kube-api-access-ksfmm\") pod \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") "
Oct 30 01:06:30.371873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.371832    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-xks-ca\") pod \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\" (UID: \"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00\") "
Oct 30 01:06:30.372126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.372099    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-openshift-ca" pod "77ce9d04-b4ba-48f3-b44f-3a37d0bfff00" (UID: "77ce9d04-b4ba-48f3-b44f-3a37d0bfff00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:30.372420 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.372398    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.374101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.374069    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-xks-ca" pod "77ce9d04-b4ba-48f3-b44f-3a37d0bfff00" (UID: "77ce9d04-b4ba-48f3-b44f-3a37d0bfff00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:30.374211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.374189    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-kube-api-access-ksfmm" pod "77ce9d04-b4ba-48f3-b44f-3a37d0bfff00" (UID: "77ce9d04-b4ba-48f3-b44f-3a37d0bfff00"). InnerVolumeSpecName "kube-api-access-ksfmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:30.473605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.473565    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksfmm\" (UniqueName: \"kubernetes.io/projected/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-kube-api-access-ksfmm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:30.473605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:30.473601    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:31.159539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:31.159497    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx" event={"ID":"77ce9d04-b4ba-48f3-b44f-3a37d0bfff00","Type":"ContainerDied","Data":"e4d99b28c3d7ef6c259a1fc02f89de466deccbe61b4bba235bc0759e5625ad9b"}
Oct 30 01:06:31.159539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:31.159524    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"
Oct 30 01:06:31.159539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:31.159545    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d99b28c3d7ef6c259a1fc02f89de466deccbe61b4bba235bc0759e5625ad9b"
Oct 30 01:06:31.635471 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:31.635433    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:32.038577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:32.038540    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"]
Oct 30 01:06:32.038758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:32.038730    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy" containerID="cri-o://a9469f134609370b0f0893d5feb5c7383e73056abf4006fbe8d3d91a6b68d069" gracePeriod=30
Oct 30 01:06:34.182666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:34.182625    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:34.619963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:34.619931    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:36.553753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:36.553720    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:36.675373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:36.675321    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:36.785204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:36.785141    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:37.203673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.203634    4576 generic.go:356] "Generic (PLEG): container finished" podID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerID="a9469f134609370b0f0893d5feb5c7383e73056abf4006fbe8d3d91a6b68d069" exitCode=0
Oct 30 01:06:37.203824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.203701    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" event={"ID":"8207408b-42d0-41ef-887c-cddea6ccfb64","Type":"ContainerDied","Data":"a9469f134609370b0f0893d5feb5c7383e73056abf4006fbe8d3d91a6b68d069"}
Oct 30 01:06:37.317811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.317746    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 01:06:37.373723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.373693    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-credential-socket\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.373916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.373825    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-envoy\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.373916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.373885    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8207408b-42d0-41ef-887c-cddea6ccfb64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8207408b-42d0-41ef-887c-cddea6ccfb64-istiod-ca-cert\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.374034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.373976    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-data\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.374034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.374019    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-kube-api-access-wp8sq\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-kube-api-access-wp8sq\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.374133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.374045    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-certs\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.374133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.374092    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-podinfo\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.374133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.374093    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-credential-socket" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:37.375694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.374467    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-data" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:37.375694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.374627    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-certs" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:37.375694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.374899    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8207408b-42d0-41ef-887c-cddea6ccfb64-istiod-ca-cert" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:37.377141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.377116    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-socket\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.377308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.377182    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-token\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-token\") pod \"8207408b-42d0-41ef-887c-cddea6ccfb64\" (UID: \"8207408b-42d0-41ef-887c-cddea6ccfb64\") "
Oct 30 01:06:37.377516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.377402    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-envoy" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:37.377516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.377441    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-kube-api-access-wp8sq" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "kube-api-access-wp8sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:37.377516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.377441    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-socket" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:37.377889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.377862    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-podinfo" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:37.378149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.378036    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:37.378149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.378054    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:37.378149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.378065    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8207408b-42d0-41ef-887c-cddea6ccfb64-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:37.378149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.378072    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:37.378149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.378080    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp8sq\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-kube-api-access-wp8sq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:37.378149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.378088    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:37.378149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.378096    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:37.378149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.378104    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8207408b-42d0-41ef-887c-cddea6ccfb64-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:37.379747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.379722    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-token" pod "8207408b-42d0-41ef-887c-cddea6ccfb64" (UID: "8207408b-42d0-41ef-887c-cddea6ccfb64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:37.479654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:37.479613    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8207408b-42d0-41ef-887c-cddea6ccfb64-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:06:38.093796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:38.093759    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:38.211814 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:38.211768    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:38.224661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:38.224631    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"
Oct 30 01:06:38.224828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:38.224655    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd" event={"ID":"8207408b-42d0-41ef-887c-cddea6ccfb64","Type":"ContainerDied","Data":"942323e7c18f030159dc03db617d7646461636af01616f8547f6665c7cd3b9c3"}
Oct 30 01:06:38.224828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:38.224727    4576 scope.go:122] "RemoveContainer" containerID="a9469f134609370b0f0893d5feb5c7383e73056abf4006fbe8d3d91a6b68d069"
Oct 30 01:06:38.301173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:38.301136    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"]
Oct 30 01:06:38.316282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:38.316212    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7d2b75-openshift-default-7d976f567-pvpjd"]
Oct 30 01:06:38.709219 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:38.709182    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:40.109671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:40.109640    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" path="/var/lib/kubelet/pods/8207408b-42d0-41ef-887c-cddea6ccfb64/volumes"
Oct 30 01:06:40.410055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:40.409961    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:41.117369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.117337    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:41.612419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.612382    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:41.724450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.724414    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:41.950632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.950598    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"]
Oct 30 01:06:41.951270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.951213    4576 cpu_manager.go:414] "containerMap: removing container" podUID="77ce9d04-b4ba-48f3-b44f-3a37d0bfff00" containerName="revoke-keys"
Oct 30 01:06:41.952317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.952284    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ce9d04-b4ba-48f3-b44f-3a37d0bfff00" containerName="revoke-keys"
Oct 30 01:06:41.952427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.952340    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy"
Oct 30 01:06:41.952427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.952350    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy"
Oct 30 01:06:41.952559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.952528    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8207408b-42d0-41ef-887c-cddea6ccfb64" containerName="istio-proxy"
Oct 30 01:06:41.952559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.952550    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ce9d04-b4ba-48f3-b44f-3a37d0bfff00" containerName="revoke-keys"
Oct 30 01:06:41.958374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.958350    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:41.978828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.978791    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"]
Oct 30 01:06:41.988721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:41.988697    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1eae0b-openshift-default-dockercfg-dvfnh\""
Oct 30 01:06:42.020227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.020178    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:42.042822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.042791    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-data\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.042996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.042909    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-kube-api-access-m8hnp\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.042996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.042968    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istiod-ca-cert\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.043110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.043018    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-socket\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.043164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.043115    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-envoy\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.043343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.043324    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-token\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.043437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.043364    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-certs\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.043437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.043403    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-podinfo\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.043533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.043440    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-credential-socket\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.158134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.158089    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-data\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.158822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.158798    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-kube-api-access-m8hnp\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.158994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.158976    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istiod-ca-cert\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.159104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.159089    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-socket\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.159252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.159216    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-envoy\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.159478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.159462    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-token\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.169072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.169032    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:42.169210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.160651    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-data\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.169210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.161351    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-socket\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.169210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.167513    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-envoy\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.180285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.160335    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istiod-ca-cert\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.180527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.180503    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-certs\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.180739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.180714    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-podinfo\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.180882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.180865    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-credential-socket\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.183262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.181372    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-credential-socket\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.183426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.181882    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-certs\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.187266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.186910    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-token\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.201376 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.192778    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-podinfo\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.201376 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.197939    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-kube-api-access-m8hnp\") pod \"e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") " pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.296261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.294043    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:42.308017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.306777    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:42.659687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.659661    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"]
Oct 30 01:06:42.660786 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:06:42.660761    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228a0ad5_07f7_43ba_bc33_a2a61a97996f.slice/crio-92a708876537a0ef3b7afbd54258164beaebbbb9b0b7d9ce8c970c88d3bad391 WatchSource:0}: Error finding container 92a708876537a0ef3b7afbd54258164beaebbbb9b0b7d9ce8c970c88d3bad391: Status 404 returned error can't find the container with id 92a708876537a0ef3b7afbd54258164beaebbbb9b0b7d9ce8c970c88d3bad391
Oct 30 01:06:42.663956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.663913    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:06:42.664066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.664006    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:06:42.664066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:42.664055    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:06:43.259380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:43.259347    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" event={"ID":"228a0ad5-07f7-43ba-bc33-a2a61a97996f","Type":"ContainerStarted","Data":"430bedfb19cc8c2e3ff10259ff2071cc4c2755e5d280c7c1a9626d95b8a0645b"}
Oct 30 01:06:43.259887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:43.259412    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" event={"ID":"228a0ad5-07f7-43ba-bc33-a2a61a97996f","Type":"ContainerStarted","Data":"92a708876537a0ef3b7afbd54258164beaebbbb9b0b7d9ce8c970c88d3bad391"}
Oct 30 01:06:43.307579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:43.307549    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:43.314024 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:43.314001    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:43.323477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:43.323432    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" podStartSLOduration=2.323418482 podStartE2EDuration="2.323418482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:43.321997661 +0000 UTC m=+1683.393321496" watchObservedRunningTime="2026-10-30 01:06:43.323418482 +0000 UTC m=+1683.394742316"
Oct 30 01:06:43.939868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:43.939833    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:44.054210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:44.054057    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:44.182280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:44.182050    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:44.269226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:44.268810    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:44.270465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:44.270443    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:06:45.734352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:45.734316    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:45.882875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:45.881071    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:46.537654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:46.537614    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:47.660826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.660795    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:47.796007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.795974    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:47.805992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.805961    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"]
Oct 30 01:06:47.810150 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.810134    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:47.836806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.836763    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"]
Oct 30 01:06:47.840586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.840563    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1eae0b-dockercfg-bn8kp\""
Oct 30 01:06:47.867310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.867222    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nnl\" (UniqueName: \"kubernetes.io/projected/5c232f16-6971-4881-89a0-63a798665351-kube-api-access-t2nnl\") pod \"payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2\" (UID: \"5c232f16-6971-4881-89a0-63a798665351\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:47.867763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.867741    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c232f16-6971-4881-89a0-63a798665351-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2\" (UID: \"5c232f16-6971-4881-89a0-63a798665351\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:47.867981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.867655    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1eae0b\""
Oct 30 01:06:47.969692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.969646    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c232f16-6971-4881-89a0-63a798665351-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2\" (UID: \"5c232f16-6971-4881-89a0-63a798665351\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:47.972220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.969931    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nnl\" (UniqueName: \"kubernetes.io/projected/5c232f16-6971-4881-89a0-63a798665351-kube-api-access-t2nnl\") pod \"payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2\" (UID: \"5c232f16-6971-4881-89a0-63a798665351\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:47.972220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:47.971016    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c232f16-6971-4881-89a0-63a798665351-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2\" (UID: \"5c232f16-6971-4881-89a0-63a798665351\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:48.002610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:48.002578    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nnl\" (UniqueName: \"kubernetes.io/projected/5c232f16-6971-4881-89a0-63a798665351-kube-api-access-t2nnl\") pod \"payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2\" (UID: \"5c232f16-6971-4881-89a0-63a798665351\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:48.124489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:48.124364    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:48.382943 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:06:48.382859    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:06:48.451188 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:48.451160    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"]
Oct 30 01:06:48.452133 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:06:48.452107    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c232f16_6971_4881_89a0_63a798665351.slice/crio-6b4d580340b0229aa0af501e729916830cb4801d5c445072c9a63b35a429493b WatchSource:0}: Error finding container 6b4d580340b0229aa0af501e729916830cb4801d5c445072c9a63b35a429493b: Status 404 returned error can't find the container with id 6b4d580340b0229aa0af501e729916830cb4801d5c445072c9a63b35a429493b
Oct 30 01:06:48.568472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:48.568441    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:49.300126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:49.300087    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2" event={"ID":"5c232f16-6971-4881-89a0-63a798665351","Type":"ContainerStarted","Data":"b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4"}
Oct 30 01:06:49.300643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:49.300147    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:06:49.300643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:49.300165    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2" event={"ID":"5c232f16-6971-4881-89a0-63a798665351","Type":"ContainerStarted","Data":"6b4d580340b0229aa0af501e729916830cb4801d5c445072c9a63b35a429493b"}
Oct 30 01:06:49.329076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:49.329046    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:49.343374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:49.343325    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2" podStartSLOduration=2.343308928 podStartE2EDuration="2.343308928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:49.342334473 +0000 UTC m=+1689.413658309" watchObservedRunningTime="2026-10-30 01:06:49.343308928 +0000 UTC m=+1689.414632763"
Oct 30 01:06:50.321303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:50.317712    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:51.769538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:51.769505    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:52.362301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:52.362265    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:53.895244 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:53.895193    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:06:53.989173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:53.989141    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:56.309161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:56.309124    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:06:58.467251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:06:58.467203    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:00.317156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:00.317128    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:07:00.487583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:00.487552    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:02.670453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:02.670422    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:03.103310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:03.103268    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"]
Oct 30 01:07:03.103507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:03.103478    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy" containerID="cri-o://ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9" gracePeriod=30
Oct 30 01:07:04.622156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:04.622126    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:06.402389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:06.402352    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:07.929177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:07.929144    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:07:07.929595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:07.929198    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:07:07.979549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:07.979518    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:08.095842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.095747    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:08.382857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.382834    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 01:07:08.419635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.419600    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:08.436221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.436191    4576 generic.go:356] "Generic (PLEG): container finished" podID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerID="ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9" exitCode=0
Oct 30 01:07:08.436221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.436260    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"
Oct 30 01:07:08.436503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.436270    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" event={"ID":"3879463c-e6a8-4a5c-92da-0ec47e7cf070","Type":"ContainerDied","Data":"ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9"}
Oct 30 01:07:08.436503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.436337    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9" event={"ID":"3879463c-e6a8-4a5c-92da-0ec47e7cf070","Type":"ContainerDied","Data":"2b208ae80119e2d49ebeb149ffe893a29e3d378444f2349081189d2677c4b0cf"}
Oct 30 01:07:08.436503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.436361    4576 scope.go:122] "RemoveContainer" containerID="ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9"
Oct 30 01:07:08.447978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.447948    4576 scope.go:122] "RemoveContainer" containerID="ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9"
Oct 30 01:07:08.448506 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:07:08.448474    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9\": container with ID starting with ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9 not found: ID does not exist" containerID="ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9"
Oct 30 01:07:08.448671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.448519    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9"} err="failed to get container status \"ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9\": rpc error: code = NotFound desc = could not find container \"ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9\": container with ID starting with ef08d0daa4972fca83584bed88dd936d9f97318cf74f36b066cb4341d53e22b9 not found: ID does not exist"
Oct 30 01:07:08.571876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.571794    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-credential-socket\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.571876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.571854    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-data\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.572108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.571906    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-podinfo\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.572108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.571955    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-kube-api-access-9vb4m\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-kube-api-access-9vb4m\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.572108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572025    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-certs\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.572294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572114    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-credential-socket" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:08.572294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572155    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istiod-ca-cert\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.572294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572184    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-data" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:08.572294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572284    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-socket\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.572509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572325    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-envoy\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.572509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572408    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-token\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-token\") pod \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\" (UID: \"3879463c-e6a8-4a5c-92da-0ec47e7cf070\") "
Oct 30 01:07:08.572509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572442    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-socket" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:08.572509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572430    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-certs" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:08.572733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.572520    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istiod-ca-cert" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:08.573284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.573179    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.573284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.573247    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.573284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.573263    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.573284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.573284    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.573591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.573298    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.574827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.574801    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-podinfo" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:08.574934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.574915    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-token" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:08.574981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.574946    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-kube-api-access-9vb4m" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "kube-api-access-9vb4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:08.575184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.575156    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-envoy" pod "3879463c-e6a8-4a5c-92da-0ec47e7cf070" (UID: "3879463c-e6a8-4a5c-92da-0ec47e7cf070"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:08.674677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.674646    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.674677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.674674    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.674677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.674683    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vb4m\" (UniqueName: \"kubernetes.io/projected/3879463c-e6a8-4a5c-92da-0ec47e7cf070-kube-api-access-9vb4m\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.674923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.674692    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3879463c-e6a8-4a5c-92da-0ec47e7cf070-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:08.790312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.790275    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"]
Oct 30 01:07:08.804500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.804456    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-qkhp9"]
Oct 30 01:07:08.810430 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.810400    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"]
Oct 30 01:07:08.811576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.811549    4576 cpu_manager.go:414] "containerMap: removing container" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy"
Oct 30 01:07:08.811746 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.811724    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy"
Oct 30 01:07:08.811920 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.811906    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" containerName="istio-proxy"
Oct 30 01:07:08.817361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.817338    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:08.837125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.833766    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2bf8a-openshift-default-dockercfg-cf8qf\""
Oct 30 01:07:08.841070 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.841025    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"]
Oct 30 01:07:08.889225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.889189    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:08.985651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:08.985604    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-envoy\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.002407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.002372    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-data\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.002699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.002678    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj7j\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-kube-api-access-klj7j\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.002822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.002808    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-credential-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.002920 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.002907    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-token\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.003025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.003011    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istiod-ca-cert\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.003292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.003275    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.003548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.003532    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-podinfo\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.003679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.003664    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-certs\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.105601    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-data\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.105682    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj7j\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-kube-api-access-klj7j\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.105718    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-credential-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.105745    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-token\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.105777    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istiod-ca-cert\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.105846    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.105884    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-podinfo\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.105921    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-certs\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.106112    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-envoy\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.106838    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-data\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.107407    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-credential-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.107639    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-socket\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.108303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.108275    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istiod-ca-cert\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.109472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.109449    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-certs\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.115280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.112748    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-envoy\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.115607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.115581    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-podinfo\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.130501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.130465    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-token\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.130712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.130596    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj7j\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-kube-api-access-klj7j\") pod \"e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") " pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.140522 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.140478    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:09.149630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.149601    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:09.418119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.417988    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:09.485921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.485891    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"]
Oct 30 01:07:09.491910 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:07:09.491874    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda890833d_89d5_40a9_a140_5e49fb5d4d7b.slice/crio-b44538a009b7314f131cdac6a12bea6782aaa3e0914e731dc3424eb9570c1934 WatchSource:0}: Error finding container b44538a009b7314f131cdac6a12bea6782aaa3e0914e731dc3424eb9570c1934: Status 404 returned error can't find the container with id b44538a009b7314f131cdac6a12bea6782aaa3e0914e731dc3424eb9570c1934
Oct 30 01:07:09.496108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.496061    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:07:09.496224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.496153    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:07:09.496224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:09.496199    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:07:10.025936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:10.025897    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:10.112143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:10.112108    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3879463c-e6a8-4a5c-92da-0ec47e7cf070" path="/var/lib/kubelet/pods/3879463c-e6a8-4a5c-92da-0ec47e7cf070/volumes"
Oct 30 01:07:10.459081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:10.459043    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" event={"ID":"a890833d-89d5-40a9-a140-5e49fb5d4d7b","Type":"ContainerStarted","Data":"e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315"}
Oct 30 01:07:10.459293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:10.459115    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" event={"ID":"a890833d-89d5-40a9-a140-5e49fb5d4d7b","Type":"ContainerStarted","Data":"b44538a009b7314f131cdac6a12bea6782aaa3e0914e731dc3424eb9570c1934"}
Oct 30 01:07:11.140941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:11.140909    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:11.146874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:11.146852    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:11.206641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:11.206593    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" podStartSLOduration=3.206579017 podStartE2EDuration="3.206579017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:10.531358157 +0000 UTC m=+1710.602681991" watchObservedRunningTime="2026-10-30 01:07:11.206579017 +0000 UTC m=+1711.277902851"
Oct 30 01:07:11.464979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:11.464942    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:11.465891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:11.465872    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:07:12.290721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:12.290683    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:13.176649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:13.176614    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:13.364889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:13.364852    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:13.497044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:13.497018    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:13.636759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:13.636722    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:13.759019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:13.758934    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:14.052208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.052109    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"]
Oct 30 01:07:14.058321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.058292    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.090314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.090276    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"]
Oct 30 01:07:14.098594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.098564    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d54608-openshift-default-dockercfg-5hshm\""
Oct 30 01:07:14.158918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.158886    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:14.185265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.185193    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-certs\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.185669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.185640    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-credential-socket\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.185790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.185737    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxkvf\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-kube-api-access-hxkvf\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.185867 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.185843    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-data\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.185926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.185893    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-envoy\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.185984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.185932    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-podinfo\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.185984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.185959    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-token\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.186097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.186056    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f069a5db-2229-42fd-9a0b-83f5e3365174-istiod-ca-cert\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.186149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.186114    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-socket\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.289164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.289118    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-credential-socket\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.289519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.289492    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxkvf\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-kube-api-access-hxkvf\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.289765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.289729    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-data\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.289941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.289924    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-envoy\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.290073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.290056    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-podinfo\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.290187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.290170    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-token\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.290380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.290365    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f069a5db-2229-42fd-9a0b-83f5e3365174-istiod-ca-cert\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.290537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.290522    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-socket\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.290704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.290687    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-certs\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.291329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.291307    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-certs\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.291690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.291673    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-credential-socket\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.292328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.292307    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-data\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.300297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.296309    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f069a5db-2229-42fd-9a0b-83f5e3365174-istiod-ca-cert\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.300297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.296616    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-socket\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.300297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.299834    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-envoy\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.312347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.312206    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-podinfo\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.343820 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.334380    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-token\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.344067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.334913    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxkvf\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-kube-api-access-hxkvf\") pod \"e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") " pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.376525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.376490    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:14.399805 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.399734    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:14.739260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.737968    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"]
Oct 30 01:07:14.740940 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:07:14.740906    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf069a5db_2229_42fd_9a0b_83f5e3365174.slice/crio-5c5fdede67fd33383ba8fbcdd81d8c6117c8963843f7ba1b3bf5cb24d31f8566 WatchSource:0}: Error finding container 5c5fdede67fd33383ba8fbcdd81d8c6117c8963843f7ba1b3bf5cb24d31f8566: Status 404 returned error can't find the container with id 5c5fdede67fd33383ba8fbcdd81d8c6117c8963843f7ba1b3bf5cb24d31f8566
Oct 30 01:07:14.744384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.744340    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:07:14.744482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.744435    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:07:14.744541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:14.744476    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:07:15.344243 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:15.344204    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:15.525143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:15.525100    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" event={"ID":"f069a5db-2229-42fd-9a0b-83f5e3365174","Type":"ContainerStarted","Data":"c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8"}
Oct 30 01:07:15.525588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:15.525171    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" event={"ID":"f069a5db-2229-42fd-9a0b-83f5e3365174","Type":"ContainerStarted","Data":"5c5fdede67fd33383ba8fbcdd81d8c6117c8963843f7ba1b3bf5cb24d31f8566"}
Oct 30 01:07:15.584125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:15.584074    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" podStartSLOduration=1.5840578189999999 podStartE2EDuration="1.584057819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:15.582597926 +0000 UTC m=+1715.653921800" watchObservedRunningTime="2026-10-30 01:07:15.584057819 +0000 UTC m=+1715.655381657"
Oct 30 01:07:16.376821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.376778    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:16.384498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.384469    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:16.532513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.532475    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:16.533460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.533441    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:07:16.639064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.638968    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:16.809903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.809872    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:16.921961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.921864    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:16.992026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.991982    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"]
Oct 30 01:07:16.996509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:16.996483    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:17.012563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.012520    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b2bf8a\""
Oct 30 01:07:17.012751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.012605    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b2bf8a-dockercfg-d4jf2\""
Oct 30 01:07:17.015078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.015055    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"]
Oct 30 01:07:17.148260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.147720    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl\" (UID: \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:17.148260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.147984    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6992x\" (UniqueName: \"kubernetes.io/projected/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-kube-api-access-6992x\") pod \"payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl\" (UID: \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:17.250209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.250118    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl\" (UID: \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:17.250726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.250644    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6992x\" (UniqueName: \"kubernetes.io/projected/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-kube-api-access-6992x\") pod \"payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl\" (UID: \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:17.251734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.251713    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl\" (UID: \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:17.272785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.272757    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6992x\" (UniqueName: \"kubernetes.io/projected/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-kube-api-access-6992x\") pod \"payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl\" (UID: \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:17.315952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.315909    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:17.663373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:17.663348    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"]
Oct 30 01:07:17.664710 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:07:17.664680    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556fdf7e_9b7d_483b_b3b6_04fa78ca3b75.slice/crio-d669cb6dee7d5708d4bdd49ef4defaca76d4572d2a6b9be6b0b7b38ab3f06e66 WatchSource:0}: Error finding container d669cb6dee7d5708d4bdd49ef4defaca76d4572d2a6b9be6b0b7b38ab3f06e66: Status 404 returned error can't find the container with id d669cb6dee7d5708d4bdd49ef4defaca76d4572d2a6b9be6b0b7b38ab3f06e66
Oct 30 01:07:18.247066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:18.247026    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:18.385805 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:07:18.385771    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:07:18.419687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:18.419655    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:18.535079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:18.534993    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:18.546595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:18.546562    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl" event={"ID":"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75","Type":"ContainerStarted","Data":"27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31"}
Oct 30 01:07:18.546738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:18.546622    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl" event={"ID":"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75","Type":"ContainerStarted","Data":"d669cb6dee7d5708d4bdd49ef4defaca76d4572d2a6b9be6b0b7b38ab3f06e66"}
Oct 30 01:07:18.546894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:18.546879    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:18.592972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:18.592922    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl" podStartSLOduration=2.592907862 podStartE2EDuration="2.592907862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:18.590221652 +0000 UTC m=+1718.661545489" watchObservedRunningTime="2026-10-30 01:07:18.592907862 +0000 UTC m=+1718.664231696"
Oct 30 01:07:18.785672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:18.785587    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:20.888206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:20.888139    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:21.827902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:21.827865    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:21.953734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:21.953699    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:21.966374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:21.966345    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"]
Oct 30 01:07:21.970260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:21.970213    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:21.985801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:21.985770    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"]
Oct 30 01:07:21.986464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:21.986441    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d54608-dockercfg-kdcc7\""
Oct 30 01:07:22.057220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.054549    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d54608\""
Oct 30 01:07:22.142563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.142448    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42l4\" (UniqueName: \"kubernetes.io/projected/a475c6d4-042d-45cd-850e-49d0eda403c5-kube-api-access-q42l4\") pod \"payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv\" (UID: \"a475c6d4-042d-45cd-850e-49d0eda403c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:22.143057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.143033    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a475c6d4-042d-45cd-850e-49d0eda403c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv\" (UID: \"a475c6d4-042d-45cd-850e-49d0eda403c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:22.244119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.244082    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42l4\" (UniqueName: \"kubernetes.io/projected/a475c6d4-042d-45cd-850e-49d0eda403c5-kube-api-access-q42l4\") pod \"payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv\" (UID: \"a475c6d4-042d-45cd-850e-49d0eda403c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:22.244313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.244264    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a475c6d4-042d-45cd-850e-49d0eda403c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv\" (UID: \"a475c6d4-042d-45cd-850e-49d0eda403c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:22.244955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.244938    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a475c6d4-042d-45cd-850e-49d0eda403c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv\" (UID: \"a475c6d4-042d-45cd-850e-49d0eda403c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:22.265167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.265141    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42l4\" (UniqueName: \"kubernetes.io/projected/a475c6d4-042d-45cd-850e-49d0eda403c5-kube-api-access-q42l4\") pod \"payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv\" (UID: \"a475c6d4-042d-45cd-850e-49d0eda403c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:22.289246 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.289198    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:22.591742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.591706    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"]
Oct 30 01:07:22.592794 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:07:22.592764    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda475c6d4_042d_45cd_850e_49d0eda403c5.slice/crio-6515a2e7799bf9b831288c6930a93cf7124ddeb9667fcbf54ac691cd84139e7a WatchSource:0}: Error finding container 6515a2e7799bf9b831288c6930a93cf7124ddeb9667fcbf54ac691cd84139e7a: Status 404 returned error can't find the container with id 6515a2e7799bf9b831288c6930a93cf7124ddeb9667fcbf54ac691cd84139e7a
Oct 30 01:07:22.967790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:22.967761    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:23.585719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:23.585682    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" event={"ID":"a475c6d4-042d-45cd-850e-49d0eda403c5","Type":"ContainerStarted","Data":"adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7"}
Oct 30 01:07:23.587080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:23.585775    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" event={"ID":"a475c6d4-042d-45cd-850e-49d0eda403c5","Type":"ContainerStarted","Data":"6515a2e7799bf9b831288c6930a93cf7124ddeb9667fcbf54ac691cd84139e7a"}
Oct 30 01:07:23.587080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:23.585886    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:23.632930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:23.632876    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" podStartSLOduration=2.632862916 podStartE2EDuration="2.632862916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:23.632756202 +0000 UTC m=+1723.704080038" watchObservedRunningTime="2026-10-30 01:07:23.632862916 +0000 UTC m=+1723.704186751"
Oct 30 01:07:24.807622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:24.807585    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:26.419117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:26.419082    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:26.537057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:26.536947    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:26.884008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:26.883977    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:27.320690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:27.320656    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:27.474849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:27.474803    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:28.408057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:28.408024    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:29.557057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:29.557026    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:07:30.086356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:30.086321    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:31.077568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:31.077533    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:31.213319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:31.213283    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:32.329401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:32.329354    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:32.448217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:32.448184    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:33.526583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:33.526543    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:33.969753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:33.969695    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:34.442946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.442661    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:34.559591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.559555    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:34.599754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.599727    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:07:34.679075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.679031    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:34.773494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.773403    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"]
Oct 30 01:07:34.777379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.777357    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:34.794604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.794576    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bc6092-dockercfg-djvvs\""
Oct 30 01:07:34.795065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.794966    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bc6092\""
Oct 30 01:07:34.795193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.795080    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"]
Oct 30 01:07:34.795301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.795202    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:34.853015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.850403    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc93b7e-19c5-4609-b9b7-4515f213f848-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm\" (UID: \"cbc93b7e-19c5-4609-b9b7-4515f213f848\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:34.853015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.850603    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbprl\" (UniqueName: \"kubernetes.io/projected/cbc93b7e-19c5-4609-b9b7-4515f213f848-kube-api-access-qbprl\") pod \"payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm\" (UID: \"cbc93b7e-19c5-4609-b9b7-4515f213f848\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:34.973271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:34.971600    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbprl\" (UniqueName: \"kubernetes.io/projected/cbc93b7e-19c5-4609-b9b7-4515f213f848-kube-api-access-qbprl\") pod \"payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm\" (UID: \"cbc93b7e-19c5-4609-b9b7-4515f213f848\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:35.041308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.034732    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc93b7e-19c5-4609-b9b7-4515f213f848-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm\" (UID: \"cbc93b7e-19c5-4609-b9b7-4515f213f848\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:35.078255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.074382    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc93b7e-19c5-4609-b9b7-4515f213f848-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm\" (UID: \"cbc93b7e-19c5-4609-b9b7-4515f213f848\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:35.110357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.096284    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"]
Oct 30 01:07:35.112106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.112063    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"]
Oct 30 01:07:35.112391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.112281    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:35.152673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.135831    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllhj\" (UniqueName: \"kubernetes.io/projected/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-kube-api-access-fllhj\") pod \"payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9\" (UID: \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:35.152673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.136386    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9\" (UID: \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:35.155445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.153950    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbprl\" (UniqueName: \"kubernetes.io/projected/cbc93b7e-19c5-4609-b9b7-4515f213f848-kube-api-access-qbprl\") pod \"payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm\" (UID: \"cbc93b7e-19c5-4609-b9b7-4515f213f848\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:35.256837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.256793    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9\" (UID: \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:35.257048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.256870    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllhj\" (UniqueName: \"kubernetes.io/projected/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-kube-api-access-fllhj\") pod \"payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9\" (UID: \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:35.257575 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.257550    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9\" (UID: \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:35.279043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.279009    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllhj\" (UniqueName: \"kubernetes.io/projected/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-kube-api-access-fllhj\") pod \"payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9\" (UID: \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:35.392656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.392558    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:35.395363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.395346    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:35.458075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.456764    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:35.704278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.704249    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"]
Oct 30 01:07:35.705598 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:07:35.705575    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc93b7e_19c5_4609_b9b7_4515f213f848.slice/crio-2fec56eead553270fe2ab6539d927830466eb062809abb8fc8fc036f096ad96a WatchSource:0}: Error finding container 2fec56eead553270fe2ab6539d927830466eb062809abb8fc8fc036f096ad96a: Status 404 returned error can't find the container with id 2fec56eead553270fe2ab6539d927830466eb062809abb8fc8fc036f096ad96a
Oct 30 01:07:35.801578 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:35.801550    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"]
Oct 30 01:07:35.802700 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:07:35.802674    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1f64ae_fe2e_4c69_8e98_aab96c76b832.slice/crio-314d9cab6c79e56f1555667fa11d7ba0fe53644473c7cd31e32888266d36d066 WatchSource:0}: Error finding container 314d9cab6c79e56f1555667fa11d7ba0fe53644473c7cd31e32888266d36d066: Status 404 returned error can't find the container with id 314d9cab6c79e56f1555667fa11d7ba0fe53644473c7cd31e32888266d36d066
Oct 30 01:07:36.191848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.189009    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:07:36.694995 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.694956    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm" event={"ID":"cbc93b7e-19c5-4609-b9b7-4515f213f848","Type":"ContainerStarted","Data":"e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522"}
Oct 30 01:07:36.695184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.695022    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm" event={"ID":"cbc93b7e-19c5-4609-b9b7-4515f213f848","Type":"ContainerStarted","Data":"2fec56eead553270fe2ab6539d927830466eb062809abb8fc8fc036f096ad96a"}
Oct 30 01:07:36.695184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.695124    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:36.697011 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.696981    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9" event={"ID":"2c1f64ae-fe2e-4c69-8e98-aab96c76b832","Type":"ContainerStarted","Data":"79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a"}
Oct 30 01:07:36.697111 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.697022    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9" event={"ID":"2c1f64ae-fe2e-4c69-8e98-aab96c76b832","Type":"ContainerStarted","Data":"314d9cab6c79e56f1555667fa11d7ba0fe53644473c7cd31e32888266d36d066"}
Oct 30 01:07:36.697182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.697124    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:36.747643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.747583    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm" podStartSLOduration=2.747565559 podStartE2EDuration="2.747565559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:36.743918563 +0000 UTC m=+1736.815242434" watchObservedRunningTime="2026-10-30 01:07:36.747565559 +0000 UTC m=+1736.818889395"
Oct 30 01:07:36.777847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.777812    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:36.796706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:36.796612    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9" podStartSLOduration=2.796596874 podStartE2EDuration="2.796596874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:36.795626246 +0000 UTC m=+1736.866950083" watchObservedRunningTime="2026-10-30 01:07:36.796596874 +0000 UTC m=+1736.867920712"
Oct 30 01:07:37.740173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:37.740132    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:38.307732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:38.307702    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:38.436985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:38.436951    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:39.055474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:39.055438    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:40.141777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.141744    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:40.390498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.390274    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:40.484329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.484292    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"]
Oct 30 01:07:40.570141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.565544    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"]
Oct 30 01:07:40.570141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.565666    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:40.583207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.583187    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-d0ae0d\""
Oct 30 01:07:40.597541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.597515    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-d0ae0d-dockercfg-rgcgw\""
Oct 30 01:07:40.625446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.625415    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6nsz\" (UniqueName: \"kubernetes.io/projected/7a984816-b004-405c-bfb9-2bb32ed59246-kube-api-access-n6nsz\") pod \"payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd\" (UID: \"7a984816-b004-405c-bfb9-2bb32ed59246\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:40.625618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.625482    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a984816-b004-405c-bfb9-2bb32ed59246-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd\" (UID: \"7a984816-b004-405c-bfb9-2bb32ed59246\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:40.726500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.726462    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6nsz\" (UniqueName: \"kubernetes.io/projected/7a984816-b004-405c-bfb9-2bb32ed59246-kube-api-access-n6nsz\") pod \"payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd\" (UID: \"7a984816-b004-405c-bfb9-2bb32ed59246\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:40.726683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.726544    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a984816-b004-405c-bfb9-2bb32ed59246-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd\" (UID: \"7a984816-b004-405c-bfb9-2bb32ed59246\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:40.727208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.727187    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a984816-b004-405c-bfb9-2bb32ed59246-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd\" (UID: \"7a984816-b004-405c-bfb9-2bb32ed59246\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:40.747199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.747122    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6nsz\" (UniqueName: \"kubernetes.io/projected/7a984816-b004-405c-bfb9-2bb32ed59246-kube-api-access-n6nsz\") pod \"payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd\" (UID: \"7a984816-b004-405c-bfb9-2bb32ed59246\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:40.889219 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:40.889190    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:41.191176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:41.191143    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"]
Oct 30 01:07:41.191736 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:07:41.191714    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a984816_b004_405c_bfb9_2bb32ed59246.slice/crio-9a7fcfae9c12f6ea3099199f11de18e6897b882c1daa593dcd44f3ad9c3346d8 WatchSource:0}: Error finding container 9a7fcfae9c12f6ea3099199f11de18e6897b882c1daa593dcd44f3ad9c3346d8: Status 404 returned error can't find the container with id 9a7fcfae9c12f6ea3099199f11de18e6897b882c1daa593dcd44f3ad9c3346d8
Oct 30 01:07:41.407623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:41.407594    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:41.737828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:41.737792    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd" event={"ID":"7a984816-b004-405c-bfb9-2bb32ed59246","Type":"ContainerStarted","Data":"19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d"}
Oct 30 01:07:41.737992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:41.737855    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd" event={"ID":"7a984816-b004-405c-bfb9-2bb32ed59246","Type":"ContainerStarted","Data":"9a7fcfae9c12f6ea3099199f11de18e6897b882c1daa593dcd44f3ad9c3346d8"}
Oct 30 01:07:41.737992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:41.737888    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:41.781592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:41.781538    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd" podStartSLOduration=1.781520231 podStartE2EDuration="1.781520231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:41.778113231 +0000 UTC m=+1741.849437068" watchObservedRunningTime="2026-10-30 01:07:41.781520231 +0000 UTC m=+1741.852844068"
Oct 30 01:07:42.971643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:42.971610    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:44.760554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:44.760518    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:46.259999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:46.259967    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:47.568487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:47.568455    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:47.712051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:47.712017    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:07:47.713610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:47.713584    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:07:48.077656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:48.077618    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:48.283334 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:48.283297    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:48.335909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:48.335812    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"]
Oct 30 01:07:48.339174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:48.339150    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c49c0c-pk6hx"]
Oct 30 01:07:48.382448 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:07:48.382412    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:07:48.412392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:48.412299    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:48.839463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:48.839425    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:50.111366 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:50.111332    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ce9d04-b4ba-48f3-b44f-3a37d0bfff00" path="/var/lib/kubelet/pods/77ce9d04-b4ba-48f3-b44f-3a37d0bfff00/volumes"
Oct 30 01:07:50.316544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:50.316507    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:51.910360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:51.910265    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:52.747936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:52.747904    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:07:53.666618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:53.666585    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:55.011086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.011053    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:55.123892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.123855    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:55.328301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.328188    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:55.692526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.692500    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:55.851392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.851351    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"]
Oct 30 01:07:55.857437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.857414    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:55.877273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.873851    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"]
Oct 30 01:07:55.877273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.873977    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:55.877273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.874347    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:07:55.942703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.942621    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:55.942703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.942699    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlgr\" (UniqueName: \"kubernetes.io/projected/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-kube-api-access-bzlgr\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:55.942905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:55.942790    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:56.044163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.044126    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlgr\" (UniqueName: \"kubernetes.io/projected/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-kube-api-access-bzlgr\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:56.044645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.044171    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:56.044645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.044369    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:56.044980 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.044960    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:56.046792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.046774    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:56.065729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.065699    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlgr\" (UniqueName: \"kubernetes.io/projected/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-kube-api-access-bzlgr\") pod \"maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:56.175745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.175712    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:56.479832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.479772    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"]
Oct 30 01:07:56.482525 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:07:56.482496    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1a3c90_2d93_4d37_8125_bda4e7cc2c81.slice/crio-3252a184ebe5cc4e9c369e1df5344c9306986b0ac771c406706884d512bfa4ce WatchSource:0}: Error finding container 3252a184ebe5cc4e9c369e1df5344c9306986b0ac771c406706884d512bfa4ce: Status 404 returned error can't find the container with id 3252a184ebe5cc4e9c369e1df5344c9306986b0ac771c406706884d512bfa4ce
Oct 30 01:07:56.874997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:56.874966    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g" event={"ID":"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81","Type":"ContainerStarted","Data":"3252a184ebe5cc4e9c369e1df5344c9306986b0ac771c406706884d512bfa4ce"}
Oct 30 01:07:57.883053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:57.883018    4576 generic.go:356] "Generic (PLEG): container finished" podID="0a1a3c90-2d93-4d37-8125-bda4e7cc2c81" containerID="a1d82415421c45c0991045fe8ae9b9afb9444e7061b241ca0d60e53864f86f62" exitCode=0
Oct 30 01:07:57.883524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:57.883107    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g" event={"ID":"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81","Type":"ContainerDied","Data":"a1d82415421c45c0991045fe8ae9b9afb9444e7061b241ca0d60e53864f86f62"}
Oct 30 01:07:57.893685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:57.893654    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:59.079701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.079673    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:59.187678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.187652    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-kube-api-access-bzlgr\" (UniqueName: \"kubernetes.io/projected/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-kube-api-access-bzlgr\") pod \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") "
Oct 30 01:07:59.187938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.187920    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-xks-ca\") pod \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") "
Oct 30 01:07:59.187999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.187981    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-openshift-ca\") pod \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\" (UID: \"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81\") "
Oct 30 01:07:59.188393 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.188370    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-openshift-ca" pod "0a1a3c90-2d93-4d37-8125-bda4e7cc2c81" (UID: "0a1a3c90-2d93-4d37-8125-bda4e7cc2c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:59.188586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.188569    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:59.189925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.189901    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-kube-api-access-bzlgr" pod "0a1a3c90-2d93-4d37-8125-bda4e7cc2c81" (UID: "0a1a3c90-2d93-4d37-8125-bda4e7cc2c81"). InnerVolumeSpecName "kube-api-access-bzlgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:59.190205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.190185    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-xks-ca" pod "0a1a3c90-2d93-4d37-8125-bda4e7cc2c81" (UID: "0a1a3c90-2d93-4d37-8125-bda4e7cc2c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:59.290294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.290261    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:59.290294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.290288    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlgr\" (UniqueName: \"kubernetes.io/projected/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81-kube-api-access-bzlgr\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:07:59.681543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.681446    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:59.873036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.872998    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:07:59.899765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.899738    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"
Oct 30 01:07:59.899765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.899762    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g" event={"ID":"0a1a3c90-2d93-4d37-8125-bda4e7cc2c81","Type":"ContainerDied","Data":"3252a184ebe5cc4e9c369e1df5344c9306986b0ac771c406706884d512bfa4ce"}
Oct 30 01:07:59.899972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:07:59.899814    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3252a184ebe5cc4e9c369e1df5344c9306986b0ac771c406706884d512bfa4ce"
Oct 30 01:08:00.150398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:00.149199    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:01.936557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:01.936521    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:03.621571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.621539    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"]
Oct 30 01:08:03.622152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.622139    4576 cpu_manager.go:414] "containerMap: removing container" podUID="0a1a3c90-2d93-4d37-8125-bda4e7cc2c81" containerName="revoke-keys"
Oct 30 01:08:03.622198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.622153    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1a3c90-2d93-4d37-8125-bda4e7cc2c81" containerName="revoke-keys"
Oct 30 01:08:03.622297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.622287    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1a3c90-2d93-4d37-8125-bda4e7cc2c81" containerName="revoke-keys"
Oct 30 01:08:03.625686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.625669    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.656400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.656348    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b483fd-openshift-default-dockercfg-rdt2m\""
Oct 30 01:08:03.659995 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.659966    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"]
Oct 30 01:08:03.708077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708036    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istiod-ca-cert\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.708308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708152    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-credential-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.708308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708248    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-certs\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.708449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708318    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-data\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.708449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708354    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-envoy\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.708449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708380    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-podinfo\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.708449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708440    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plmf\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-kube-api-access-9plmf\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.708666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708574    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-token\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.708666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.708606    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.762642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.762608    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:03.814202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.814160    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-data\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.814442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.814222    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-envoy\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.814442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.814306    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-podinfo\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.814553 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.814460    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plmf\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-kube-api-access-9plmf\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.814668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.814645    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-token\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.815104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.814926    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.815380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.815358    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istiod-ca-cert\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.815560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.815543    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-credential-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.815688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.815675    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-certs\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.816326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.816290    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-certs\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.816894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.816864    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-data\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.817096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.817070    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.817915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.817890    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-credential-socket\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.818037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.818017    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istiod-ca-cert\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.820369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.820347    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-envoy\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.821204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.820753    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-podinfo\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.837891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.837853    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plmf\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-kube-api-access-9plmf\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.838426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.838404    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-token\") pod \"e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") " pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:03.941017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:03.940991    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:04.130039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.128830    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"]
Oct 30 01:08:04.130039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.128883    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"]
Oct 30 01:08:04.130039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.129086    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy" containerID="cri-o://dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5" gracePeriod=30
Oct 30 01:08:04.130039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.129600    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy" containerID="cri-o://8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1" gracePeriod=30
Oct 30 01:08:04.297364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.295586    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"]
Oct 30 01:08:04.300330 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:08:04.299985    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96b3f6f_3e9b_495f_bde2_1c83f9b6753e.slice/crio-c2032500a8f1f2f30ae5e84b43465abdc12d1bd9a984807b9f0f2c9ad8005ccb WatchSource:0}: Error finding container c2032500a8f1f2f30ae5e84b43465abdc12d1bd9a984807b9f0f2c9ad8005ccb: Status 404 returned error can't find the container with id c2032500a8f1f2f30ae5e84b43465abdc12d1bd9a984807b9f0f2c9ad8005ccb
Oct 30 01:08:04.303334 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.303303    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:04.303434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.303373    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:04.303434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.303405    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:04.939509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.939472    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" event={"ID":"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e","Type":"ContainerStarted","Data":"f5c435eb23cfda2b16f84e9720920a2d30b1e05d3da2d278037f3e1d0483a6ea"}
Oct 30 01:08:04.940082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.940060    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" event={"ID":"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e","Type":"ContainerStarted","Data":"c2032500a8f1f2f30ae5e84b43465abdc12d1bd9a984807b9f0f2c9ad8005ccb"}
Oct 30 01:08:04.941512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.941483    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:04.999919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:04.996634    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podStartSLOduration=1.996613139 podStartE2EDuration="1.996613139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:04.995291748 +0000 UTC m=+1765.066615585" watchObservedRunningTime="2026-10-30 01:08:04.996613139 +0000 UTC m=+1765.067936975"
Oct 30 01:08:05.422316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:05.415789    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:05.537260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:05.536571    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:05.834106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:05.834068    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:05.943904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:05.943732    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:08:05.943904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:05.943814    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 01:08:06.164310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:06.163883    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:06.339258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:06.338423    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:06.944260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:06.941827    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:08:06.944260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:06.941901    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 01:08:07.365905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.365815    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:07.639731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.639650    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"]
Oct 30 01:08:07.646474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.646447    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.652977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.652926    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"]
Oct 30 01:08:07.666469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.666438    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ca20f-openshift-default-dockercfg-z7jsf\""
Oct 30 01:08:07.759313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.759286    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-credential-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.759586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.759557    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-istio-token\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.759709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.759693    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36025db8-f787-4abf-be74-d6aab77bc769-istio-podinfo\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.759862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.759850    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vh7s\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-kube-api-access-4vh7s\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.760432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.760057    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-certs\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.760432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.760124    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-data\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.760432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.760173    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-envoy\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.760432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.760255    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.760432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.760282    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36025db8-f787-4abf-be74-d6aab77bc769-istiod-ca-cert\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.860985    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vh7s\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-kube-api-access-4vh7s\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861088    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-certs\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861117    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-data\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861141    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-envoy\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861178    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861207    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36025db8-f787-4abf-be74-d6aab77bc769-istiod-ca-cert\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861309    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-credential-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861367    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-istio-token\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.861489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861424    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36025db8-f787-4abf-be74-d6aab77bc769-istio-podinfo\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.862041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861621    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-certs\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.862041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861699    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-data\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.862041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.861985    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36025db8-f787-4abf-be74-d6aab77bc769-istiod-ca-cert\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.862203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.862131    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.862203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.862194    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-credential-socket\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.863873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.863855    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-envoy\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.864090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.864073    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36025db8-f787-4abf-be74-d6aab77bc769-istio-podinfo\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.900582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.900521    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-istio-token\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.900836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.900814    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vh7s\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-kube-api-access-4vh7s\") pod \"e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") " pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:07.941623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.941593    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:08:07.941792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.941645    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 01:08:07.995321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:07.995284    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:08.334208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:08.334137    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"]
Oct 30 01:08:08.335171 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:08:08.335144    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36025db8_f787_4abf_be74_d6aab77bc769.slice/crio-8a4473a70dba938e0e63475bf2f8762f833b2d365a76dd08f8a04b809aadaf2e WatchSource:0}: Error finding container 8a4473a70dba938e0e63475bf2f8762f833b2d365a76dd08f8a04b809aadaf2e: Status 404 returned error can't find the container with id 8a4473a70dba938e0e63475bf2f8762f833b2d365a76dd08f8a04b809aadaf2e
Oct 30 01:08:08.338613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:08.338581    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:08.338712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:08.338646    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:08.338712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:08.338682    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:08.941579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:08.941542    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:08:08.941791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:08.941607    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 01:08:08.961849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:08.961819    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:09.075258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.066368    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" event={"ID":"36025db8-f787-4abf-be74-d6aab77bc769","Type":"ContainerStarted","Data":"eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870"}
Oct 30 01:08:09.075258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.066445    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" event={"ID":"36025db8-f787-4abf-be74-d6aab77bc769","Type":"ContainerStarted","Data":"8a4473a70dba938e0e63475bf2f8762f833b2d365a76dd08f8a04b809aadaf2e"}
Oct 30 01:08:09.140433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.139286    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" podStartSLOduration=2.13926447 podStartE2EDuration="2.13926447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:09.120711346 +0000 UTC m=+1769.192035182" watchObservedRunningTime="2026-10-30 01:08:09.13926447 +0000 UTC m=+1769.210588307"
Oct 30 01:08:09.545288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.545262    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:08:09.643146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.643109    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istiod-ca-cert\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.643431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.643410    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-podinfo\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.643597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.643583    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-socket\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.643716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.643701    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-credential-socket\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.643846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.643829    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-token\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-token\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.643984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.643969    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-certs\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.644170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.644153    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-kube-api-access-rcpgn\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-kube-api-access-rcpgn\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.644366 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.644348    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-envoy\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.644518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.644503    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-data\") pod \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\" (UID: \"9f90a5b9-4d16-48c1-90eb-b685a04942f3\") "
Oct 30 01:08:09.645402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.644524    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-credential-socket" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:09.645504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.644687    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-socket" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:09.646960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.646928    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-podinfo" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:09.647448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.647293    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istiod-ca-cert" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:09.647448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.647394    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-data" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:09.647615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.647598    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-certs" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:09.647969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.647787    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-envoy" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:09.650911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.650858    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-token" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:09.651930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.651899    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-kube-api-access-rcpgn" pod "9f90a5b9-4d16-48c1-90eb-b685a04942f3" (UID: "9f90a5b9-4d16-48c1-90eb-b685a04942f3"). InnerVolumeSpecName "kube-api-access-rcpgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747179    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747210    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747223    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747255    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747267    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747279    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747291    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747304    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f90a5b9-4d16-48c1-90eb-b685a04942f3-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.747779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.747317    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcpgn\" (UniqueName: \"kubernetes.io/projected/9f90a5b9-4d16-48c1-90eb-b685a04942f3-kube-api-access-rcpgn\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:09.945126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.943268    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:08:09.945126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.943326    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 01:08:09.963933 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:09.963904    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:08:10.012262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.008428    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:10.030331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.028610    4576 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:08:10.030331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.028663    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052182    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-kube-api-access-4g64t\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-kube-api-access-4g64t\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052318    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-certs\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052367    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-token\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-token\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052391    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-podinfo\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052418    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-socket\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052448    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-credential-socket\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052481    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istiod-ca-cert\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052536    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-envoy\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.052633    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-data\") pod \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\" (UID: \"e5dcef84-ed5d-44df-b042-aa36913c4cd4\") "
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.053111    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istiod-ca-cert" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.053345    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-socket" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.053503    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-certs" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.053585    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.053936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.053609    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.055743 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.055556    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-credential-socket" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:10.055743 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.055711    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-kube-api-access-4g64t" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "kube-api-access-4g64t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:10.146173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.146131    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-data" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:10.153055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.153012    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-podinfo" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:10.159542 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.159506    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.159542 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.159545    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g64t\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-kube-api-access-4g64t\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.159801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.159562    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.159801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.159586    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.159801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.159599    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.159801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.159638    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-envoy" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:10.161372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.161339    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-token" pod "e5dcef84-ed5d-44df-b042-aa36913c4cd4" (UID: "e5dcef84-ed5d-44df-b042-aa36913c4cd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:10.162124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.162095    4576 generic.go:356] "Generic (PLEG): container finished" podID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerID="dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5" exitCode=0
Oct 30 01:08:10.166345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.166315    4576 generic.go:356] "Generic (PLEG): container finished" podID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerID="8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1" exitCode=0
Oct 30 01:08:10.166802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.166781    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"
Oct 30 01:08:10.223424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.168932    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" event={"ID":"e5dcef84-ed5d-44df-b042-aa36913c4cd4","Type":"ContainerDied","Data":"dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5"}
Oct 30 01:08:10.223718 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.223681    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk" event={"ID":"e5dcef84-ed5d-44df-b042-aa36913c4cd4","Type":"ContainerDied","Data":"ba307b22e288282959751f2b7fe69f9b9eefde93eb6031247356692080f2cd65"}
Oct 30 01:08:10.223901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.223879    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" event={"ID":"9f90a5b9-4d16-48c1-90eb-b685a04942f3","Type":"ContainerDied","Data":"8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1"}
Oct 30 01:08:10.224044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.224026    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz" event={"ID":"9f90a5b9-4d16-48c1-90eb-b685a04942f3","Type":"ContainerDied","Data":"f42f74954f9f58c5c194b883c8f3f088cc561f46bf532199be2966f29cdae5be"}
Oct 30 01:08:10.224155 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.223950    4576 scope.go:122] "RemoveContainer" containerID="dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5"
Oct 30 01:08:10.227537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.196282    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"
Oct 30 01:08:10.246774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.246751    4576 scope.go:122] "RemoveContainer" containerID="dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5"
Oct 30 01:08:10.247140 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:08:10.247113    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5\": container with ID starting with dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5 not found: ID does not exist" containerID="dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5"
Oct 30 01:08:10.247224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.247152    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5"} err="failed to get container status \"dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5\": rpc error: code = NotFound desc = could not find container \"dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5\": container with ID starting with dbe5dffb31a4790e1704cbd572e60a00f5f374895a66a2f4d4e980dbc276d4b5 not found: ID does not exist"
Oct 30 01:08:10.247224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.247174    4576 scope.go:122] "RemoveContainer" containerID="8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1"
Oct 30 01:08:10.259870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.259847    4576 scope.go:122] "RemoveContainer" containerID="8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1"
Oct 30 01:08:10.260412 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:08:10.260381    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1\": container with ID starting with 8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1 not found: ID does not exist" containerID="8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1"
Oct 30 01:08:10.260549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.260528    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1"} err="failed to get container status \"8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1\": rpc error: code = NotFound desc = could not find container \"8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1\": container with ID starting with 8338aad1df933a3f04d4c6d6d22dac0ff09d5396583caed1e0d79890989a53e1 not found: ID does not exist"
Oct 30 01:08:10.260853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.260828    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.260953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.260858    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dcef84-ed5d-44df-b042-aa36913c4cd4-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:10.323539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.323450    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"]
Oct 30 01:08:10.328105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.328074    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-njftz"]
Oct 30 01:08:10.328840 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:08:10.328805    4576 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dcef84_ed5d_44df_b042_aa36913c4cd4.slice/crio-ba307b22e288282959751f2b7fe69f9b9eefde93eb6031247356692080f2cd65\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f90a5b9_4d16_48c1_90eb_b685a04942f3.slice/crio-f42f74954f9f58c5c194b883c8f3f088cc561f46bf532199be2966f29cdae5be\": RecentStats: unable to find data in memory cache]"
Oct 30 01:08:10.352759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.352727    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"]
Oct 30 01:08:10.355831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.355805    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zkldk"]
Oct 30 01:08:10.459312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.459270    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:10.941596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.941556    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 01:08:10.941773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.941630    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 01:08:10.996174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.995674    4576 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:08:10.996174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:10.995736    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 01:08:11.946472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:11.946442    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:11.947045 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:11.946951    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:11.947691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:11.947668    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:08:11.995934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:11.995892    4576 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:08:11.996122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:11.995950    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 01:08:12.124881 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:12.124844    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" path="/var/lib/kubelet/pods/9f90a5b9-4d16-48c1-90eb-b685a04942f3/volumes"
Oct 30 01:08:12.125555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:12.125534    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" path="/var/lib/kubelet/pods/e5dcef84-ed5d-44df-b042-aa36913c4cd4/volumes"
Oct 30 01:08:12.221904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:12.221816    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:12.996105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:12.996066    4576 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:08:12.996539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:12.996144    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 01:08:13.793930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:13.793837    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:13.995944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:13.995902    4576 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 01:08:13.996153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:13.995977    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 01:08:14.242306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:14.242268    4576 generic.go:356] "Generic (PLEG): container finished" podID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerID="430bedfb19cc8c2e3ff10259ff2071cc4c2755e5d280c7c1a9626d95b8a0645b" exitCode=0
Oct 30 01:08:14.242481 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:14.242330    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" event={"ID":"228a0ad5-07f7-43ba-bc33-a2a61a97996f","Type":"ContainerDied","Data":"430bedfb19cc8c2e3ff10259ff2071cc4c2755e5d280c7c1a9626d95b8a0645b"}
Oct 30 01:08:14.242779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:14.242755    4576 scope.go:122] "RemoveContainer" containerID="430bedfb19cc8c2e3ff10259ff2071cc4c2755e5d280c7c1a9626d95b8a0645b"
Oct 30 01:08:14.244612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:14.244586    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:14.244671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:14.244657    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:14.244707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:14.244689    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:14.293069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:14.292324    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:08:15.021703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:15.021673    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:15.022956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:15.022934    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:15.023841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:15.023820    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:08:15.250615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:15.250578    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" event={"ID":"228a0ad5-07f7-43ba-bc33-a2a61a97996f","Type":"ContainerStarted","Data":"1e8537b88381ab1281353fc9c007c88710baa34e7bf34e342afc56fa324d619d"}
Oct 30 01:08:15.308357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:15.308262    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:08:15.309769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:15.309742    4576 patch_prober.go:29] interesting pod/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:08:15.309910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:15.309797    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 01:08:15.330153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:15.330117    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:16.308321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:16.308282    4576 patch_prober.go:29] interesting pod/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:08:16.308725 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:16.308335    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 01:08:17.034300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:17.034261    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:17.186259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:17.177819    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:17.308047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:17.307971    4576 patch_prober.go:29] interesting pod/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:08:17.308047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:17.308026    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 01:08:18.308178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:18.308143    4576 patch_prober.go:29] interesting pod/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:08:18.308612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:18.308208    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 01:08:18.410107 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:08:18.410074    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:08:18.766132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:18.766080    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:19.308212 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:19.308174    4576 patch_prober.go:29] interesting pod/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:08:19.308654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:19.308280    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 01:08:20.308010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:20.307977    4576 patch_prober.go:29] interesting pod/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 01:08:20.308204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:20.308050    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 01:08:20.363784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:20.363739    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:21.312447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:21.312418    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:08:21.312785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:21.312763    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:08:21.313591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:21.313569    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:08:22.299529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:22.299493    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:22.759809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:22.759780    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:23.997013 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:23.996979    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"]
Oct 30 01:08:23.997635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:23.997619    4576 cpu_manager.go:414] "containerMap: removing container" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy"
Oct 30 01:08:23.997697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:23.997637    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy"
Oct 30 01:08:23.997697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:23.997681    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy"
Oct 30 01:08:23.997697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:23.997687    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy"
Oct 30 01:08:23.997829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:23.997815    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5dcef84-ed5d-44df-b042-aa36913c4cd4" containerName="istio-proxy"
Oct 30 01:08:23.997875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:23.997833    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f90a5b9-4d16-48c1-90eb-b685a04942f3" containerName="istio-proxy"
Oct 30 01:08:24.002698 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.002673    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.023940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.023903    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"]
Oct 30 01:08:24.032675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.032638    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49d9f1\"/\"default-dockercfg-9fplf\""
Oct 30 01:08:24.032861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.032707    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49d9f1\"/\"openshift-service-ca.crt\""
Oct 30 01:08:24.032861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.032721    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49d9f1\"/\"kube-root-ca.crt\""
Oct 30 01:08:24.033297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.033274    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49d9f1\"/\"test-model-49d9f1-kserve-self-signed-certs\""
Oct 30 01:08:24.077761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.077722    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:24.114187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.114151    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tmp-dir\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.114392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.114203    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-dshm\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.114392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.114369    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-home\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.114523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.114421    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tls-certs\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.114680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.114645    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cn59\" (UniqueName: \"kubernetes.io/projected/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-kube-api-access-4cn59\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.114790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.114718    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-model-cache\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.216382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.216345    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-dshm\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.216582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.216428    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-home\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.216582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.216467    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tls-certs\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.216710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.216628    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cn59\" (UniqueName: \"kubernetes.io/projected/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-kube-api-access-4cn59\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.216710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.216684    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-model-cache\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.216828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.216776    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tmp-dir\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.217270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.217227    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-home\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.217692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.217471    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-model-cache\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.217692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.217622    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tmp-dir\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.219136 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.219116    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-dshm\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.219716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.219697    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tls-certs\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.244261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.244092    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cn59\" (UniqueName: \"kubernetes.io/projected/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-kube-api-access-4cn59\") pod \"test-model-49d9f1-kserve-7476f54b8f-k9q8p\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") " pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.277899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.277804    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:24.317719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.317675    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:24.689276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:24.689052    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"]
Oct 30 01:08:25.346291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:25.346221    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p" event={"ID":"a2a41a95-372b-4cc6-99f4-0b1c94325a4f","Type":"ContainerStarted","Data":"bd65f5acf0249bfbf779c47f96a7b2fddab283a3a9623076956b35bd05049709"}
Oct 30 01:08:25.346706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:25.346299    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p" event={"ID":"a2a41a95-372b-4cc6-99f4-0b1c94325a4f","Type":"ContainerStarted","Data":"13e06e4b893c8f4bcbff7e60d2f059544705fdea3e78e6c5f2db4267ff65a581"}
Oct 30 01:08:25.390796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:25.390739    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p" podStartSLOduration=2.390721415 podStartE2EDuration="2.390721415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:25.389480198 +0000 UTC m=+1785.460804035" watchObservedRunningTime="2026-10-30 01:08:25.390721415 +0000 UTC m=+1785.462045254"
Oct 30 01:08:25.501506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:25.501453    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:08:26.039087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:26.039051    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:26.152125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:26.152092    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:26.318875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:26.318790    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:28.097258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:28.097037    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:29.798156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:29.798123    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:31.301256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:31.297797    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:32.751843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:32.751809    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:34.319103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:34.319067    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:34.333680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:34.333649    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:34.466065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:34.466034    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:08:34.484574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:34.484536    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:34.914954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:34.914919    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:35.917849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:35.917806    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:36.043915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:36.043875    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:36.162418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:36.162373    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:36.523158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:36.523110    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:37.668893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:37.668820    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:38.483038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.483003    4576 generic.go:356] "Generic (PLEG): container finished" podID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerID="e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315" exitCode=0
Oct 30 01:08:38.483326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.483079    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" event={"ID":"a890833d-89d5-40a9-a140-5e49fb5d4d7b","Type":"ContainerDied","Data":"e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315"}
Oct 30 01:08:38.483628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.483607    4576 scope.go:122] "RemoveContainer" containerID="e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315"
Oct 30 01:08:38.484864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.484846    4576 generic.go:356] "Generic (PLEG): container finished" podID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerID="c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8" exitCode=0
Oct 30 01:08:38.484964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.484883    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" event={"ID":"f069a5db-2229-42fd-9a0b-83f5e3365174","Type":"ContainerDied","Data":"c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8"}
Oct 30 01:08:38.485285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.485271    4576 scope.go:122] "RemoveContainer" containerID="c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8"
Oct 30 01:08:38.485638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.485613    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:38.485691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.485678    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:38.485737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.485717    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:38.487038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.487009    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:38.487102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.487076    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:38.487137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.487121    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:08:38.872218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:38.872128    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:39.365041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:39.365014    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:39.536294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:39.535393    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" event={"ID":"f069a5db-2229-42fd-9a0b-83f5e3365174","Type":"ContainerStarted","Data":"4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f"}
Oct 30 01:08:39.539515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:39.539416    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" event={"ID":"a890833d-89d5-40a9-a140-5e49fb5d4d7b","Type":"ContainerStarted","Data":"0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d"}
Oct 30 01:08:40.141613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:40.141530    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:08:40.142747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:40.142728    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:08:40.142829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:40.142770    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 01:08:40.391260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:40.390870    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:08:40.391260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:40.390994    4576 patch_prober.go:29] interesting pod/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 01:08:40.391260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:40.391035    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 01:08:40.983107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:40.983049    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:41.141961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:41.141931    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:08:41.142388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:41.141981    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 01:08:41.381309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:41.380613    4576 patch_prober.go:29] interesting pod/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 01:08:41.381309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:41.380672    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 01:08:41.439304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:41.439270    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:41.465429 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:41.465402    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:08:41.553944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:41.553906    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:42.142032 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:42.141980    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:08:42.142555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:42.142043    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 01:08:42.380688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:42.380649    4576 patch_prober.go:29] interesting pod/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 01:08:42.380964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:42.380935    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 01:08:42.504098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:42.504064    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:43.141736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:43.141701    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:08:43.142005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:43.141756    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 01:08:43.383700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:43.383659    4576 patch_prober.go:29] interesting pod/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 01:08:43.384354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:43.384321    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 01:08:44.141079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:44.141044    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 01:08:44.141332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:44.141104    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 01:08:44.165619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:44.165588    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:44.352130 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:44.352087    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:44.385264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:44.384711    4576 patch_prober.go:29] interesting pod/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 01:08:44.385264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:44.384785    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 01:08:44.493262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:44.492750    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:45.146603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:45.146576    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:08:45.147785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:45.147756    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:08:45.382688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:45.382655    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:08:45.382951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:45.382927    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:08:45.383544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:45.383519    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:08:45.798772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:45.798727    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:47.579530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:47.579493    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:47.799196 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:47.799165    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"]
Oct 30 01:08:47.799436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:47.799411    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy" containerID="cri-o://a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64" gracePeriod=30
Oct 30 01:08:48.062604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:48.062568    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:48.276680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:48.276643    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"]
Oct 30 01:08:48.277171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:48.277139    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy" containerID="cri-o://c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba" gracePeriod=30
Oct 30 01:08:48.436017 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:08:48.435938    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:08:49.638624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:49.638585    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:50.186977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:50.186948    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:51.678023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:51.677981    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:53.086554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.086489    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 01:08:53.201357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.201269    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-data\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.201357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.201363    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-token\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-token\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.201685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.201481    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e33b94-880e-4145-9cf6-97b89a72b5b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e33b94-880e-4145-9cf6-97b89a72b5b3-istiod-ca-cert\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.201685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.201530    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-credential-socket\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.201685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.201576    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-data" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.201685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.201646    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-kube-api-access-wsqxp\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-kube-api-access-wsqxp\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.201913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.201875    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-certs\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.201913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.201875    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-credential-socket" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.202221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.202154    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e33b94-880e-4145-9cf6-97b89a72b5b3-istiod-ca-cert" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:53.202658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.202627    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-certs" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.202761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.202712    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-podinfo\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.202816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.202801    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-socket\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.202854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.202841    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-envoy\") pod \"44e33b94-880e-4145-9cf6-97b89a72b5b3\" (UID: \"44e33b94-880e-4145-9cf6-97b89a72b5b3\") "
Oct 30 01:08:53.203176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.203154    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-socket" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.203790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.203674    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44e33b94-880e-4145-9cf6-97b89a72b5b3-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.203790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.203706    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.203790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.203720    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.203790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.203735    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.203790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.203748    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.204407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.204382    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-kube-api-access-wsqxp" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "kube-api-access-wsqxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:53.204781 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.204760    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-token" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:53.205682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.205662    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-envoy" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.206379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.206358    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-podinfo" pod "44e33b94-880e-4145-9cf6-97b89a72b5b3" (UID: "44e33b94-880e-4145-9cf6-97b89a72b5b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:53.305650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.305618    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.305650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.305645    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsqxp\" (UniqueName: \"kubernetes.io/projected/44e33b94-880e-4145-9cf6-97b89a72b5b3-kube-api-access-wsqxp\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.305806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.305661    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.305806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.305669    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44e33b94-880e-4145-9cf6-97b89a72b5b3-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.348970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.348888    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:53.542604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.542570    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:53.613804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.613760    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 01:08:53.692787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.692748    4576 generic.go:356] "Generic (PLEG): container finished" podID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerID="a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64" exitCode=0
Oct 30 01:08:53.692957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.692848    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" event={"ID":"44e33b94-880e-4145-9cf6-97b89a72b5b3","Type":"ContainerDied","Data":"a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64"}
Oct 30 01:08:53.692957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.692901    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8" event={"ID":"44e33b94-880e-4145-9cf6-97b89a72b5b3","Type":"ContainerDied","Data":"1fb5bd0c33f07380a2aed5534352d5fbff734675d770f2f360c3592ee42a7ee8"}
Oct 30 01:08:53.692957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.692931    4576 scope.go:122] "RemoveContainer" containerID="a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64"
Oct 30 01:08:53.693101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.693088    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"
Oct 30 01:08:53.694886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.694854    4576 generic.go:356] "Generic (PLEG): container finished" podID="74987a39-7f9e-4389-80c4-c166af317c0a" containerID="c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba" exitCode=0
Oct 30 01:08:53.695030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.694910    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"
Oct 30 01:08:53.695181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.695114    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" event={"ID":"74987a39-7f9e-4389-80c4-c166af317c0a","Type":"ContainerDied","Data":"c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba"}
Oct 30 01:08:53.695181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.695158    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt" event={"ID":"74987a39-7f9e-4389-80c4-c166af317c0a","Type":"ContainerDied","Data":"d8b7580f77604ffc38cdc6392180eda4addf5de4b3785ddf6205f64563e42266"}
Oct 30 01:08:53.706480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.706376    4576 scope.go:122] "RemoveContainer" containerID="a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64"
Oct 30 01:08:53.707137 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:08:53.706982    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64\": container with ID starting with a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64 not found: ID does not exist" containerID="a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64"
Oct 30 01:08:53.707137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.707021    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64"} err="failed to get container status \"a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64\": rpc error: code = NotFound desc = could not find container \"a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64\": container with ID starting with a44b3f92cf2718bb54c7541036e6710bc63abb44557004980ab7335d7004eb64 not found: ID does not exist"
Oct 30 01:08:53.707137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.707045    4576 scope.go:122] "RemoveContainer" containerID="c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba"
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.712706    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-socket\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.712770    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-credential-socket\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.712837    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74987a39-7f9e-4389-80c4-c166af317c0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74987a39-7f9e-4389-80c4-c166af317c0a-istio-podinfo\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.712888    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-certs\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.712921    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-data\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.713324    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-credential-socket" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.713560    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-socket" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.713863    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-istio-token\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-istio-token\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.713913    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-envoy\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.713991    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74987a39-7f9e-4389-80c4-c166af317c0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74987a39-7f9e-4389-80c4-c166af317c0a-istiod-ca-cert\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.714044    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-kube-api-access-t2g72\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-kube-api-access-t2g72\") pod \"74987a39-7f9e-4389-80c4-c166af317c0a\" (UID: \"74987a39-7f9e-4389-80c4-c166af317c0a\") "
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.714816    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.714839    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.715987    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-data" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.716085    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-certs" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.716384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.716343    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74987a39-7f9e-4389-80c4-c166af317c0a-istiod-ca-cert" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:53.717558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.717386    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74987a39-7f9e-4389-80c4-c166af317c0a-istio-podinfo" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:53.719061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.719039    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-istio-token" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:53.719800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.719781    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-kube-api-access-t2g72" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "kube-api-access-t2g72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:53.726648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.724261    4576 scope.go:122] "RemoveContainer" containerID="c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba"
Oct 30 01:08:53.726648 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:08:53.725704    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba\": container with ID starting with c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba not found: ID does not exist" containerID="c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba"
Oct 30 01:08:53.726648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.725737    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba"} err="failed to get container status \"c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba\": rpc error: code = NotFound desc = could not find container \"c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba\": container with ID starting with c28e1321f443a747e70d1fedd39737eef7dfab2e0cc8eea0b95b53dcadaf4eba not found: ID does not exist"
Oct 30 01:08:53.729885 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.729860    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-envoy" pod "74987a39-7f9e-4389-80c4-c166af317c0a" (UID: "74987a39-7f9e-4389-80c4-c166af317c0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:53.767149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.767113    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"]
Oct 30 01:08:53.789376 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.788996    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-sl6f8"]
Oct 30 01:08:53.850626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.830528    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74987a39-7f9e-4389-80c4-c166af317c0a-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.850626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.830561    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.850626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.830576    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.850626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.830588    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.850626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.830600    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74987a39-7f9e-4389-80c4-c166af317c0a-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.850626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.830612    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74987a39-7f9e-4389-80c4-c166af317c0a-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:53.850626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:53.830625    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2g72\" (UniqueName: \"kubernetes.io/projected/74987a39-7f9e-4389-80c4-c166af317c0a-kube-api-access-t2g72\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:08:54.046822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:54.046786    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"]
Oct 30 01:08:54.055655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:54.055622    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-scfpt"]
Oct 30 01:08:54.109579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:54.109544    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" path="/var/lib/kubelet/pods/44e33b94-880e-4145-9cf6-97b89a72b5b3/volumes"
Oct 30 01:08:54.110012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:54.109997    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" path="/var/lib/kubelet/pods/74987a39-7f9e-4389-80c4-c166af317c0a/volumes"
Oct 30 01:08:54.708116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:54.708084    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:54.811731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:54.811697    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:55.121204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:55.121113    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:55.241586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:55.241551    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:56.688650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:56.688608    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:58.387266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:58.387195    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:08:59.947559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:08:59.947520    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:01.558377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:01.558338    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:03.574250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:03.574203    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:05.184811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:05.184775    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:06.705493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:06.705455    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:09:06.763543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:06.763503    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:08.140990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:08.140953    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:08.699572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:08.699529    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:10.472932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:10.472900    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:12.345635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:12.345597    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:14.367114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:14.367078    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:15.348996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.348963    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:15.588151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.588117    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"]
Oct 30 01:09:15.588958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.588937    4576 cpu_manager.go:414] "containerMap: removing container" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy"
Oct 30 01:09:15.589124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.588960    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy"
Oct 30 01:09:15.589124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.588981    4576 cpu_manager.go:414] "containerMap: removing container" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy"
Oct 30 01:09:15.589124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.588988    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy"
Oct 30 01:09:15.589346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.589193    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e33b94-880e-4145-9cf6-97b89a72b5b3" containerName="istio-proxy"
Oct 30 01:09:15.589346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.589221    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="74987a39-7f9e-4389-80c4-c166af317c0a" containerName="istio-proxy"
Oct 30 01:09:15.593088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.593070    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.610065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.610036    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"]
Oct 30 01:09:15.620451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.620413    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2bf8a\"/\"default-dockercfg-2s4jq\""
Oct 30 01:09:15.620589 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.620466    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b2bf8a\"/\"test-model-b2bf8a-kserve-self-signed-certs\""
Oct 30 01:09:15.623445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.623425    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2bf8a\"/\"openshift-service-ca.crt\""
Oct 30 01:09:15.623580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.623488    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b2bf8a\"/\"kube-root-ca.crt\""
Oct 30 01:09:15.653065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.653035    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:15.664182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.664152    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tls-certs\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.664394 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.664310    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdft\" (UniqueName: \"kubernetes.io/projected/c4587c86-fe98-40d7-8473-30f6fb0f79dd-kube-api-access-2tdft\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.664455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.664413    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-home\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.664455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.664441    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-dshm\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.664625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.664613    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tmp-dir\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.664661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.664641    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-model-cache\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766341    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tmp-dir\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766382    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-model-cache\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766408    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tls-certs\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766489    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdft\" (UniqueName: \"kubernetes.io/projected/c4587c86-fe98-40d7-8473-30f6fb0f79dd-kube-api-access-2tdft\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766555    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-home\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766577    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-dshm\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766839    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-model-cache\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766882    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tmp-dir\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.766972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.766934    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-home\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.769205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.769176    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-dshm\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.770149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.770124    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tls-certs\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.789433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.789383    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdft\" (UniqueName: \"kubernetes.io/projected/c4587c86-fe98-40d7-8473-30f6fb0f79dd-kube-api-access-2tdft\") pod \"test-model-b2bf8a-kserve-75d7b44465-d2qhr\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") " pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:15.906728 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:15.906637    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:16.227868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:16.227840    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"]
Oct 30 01:09:16.229447 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:16.229407    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4587c86_fe98_40d7_8473_30f6fb0f79dd.slice/crio-3ad4dd065362df4aeac8e69e007e4c915f9005b25695dcbd29bdf7fc8a8254ee WatchSource:0}: Error finding container 3ad4dd065362df4aeac8e69e007e4c915f9005b25695dcbd29bdf7fc8a8254ee: Status 404 returned error can't find the container with id 3ad4dd065362df4aeac8e69e007e4c915f9005b25695dcbd29bdf7fc8a8254ee
Oct 30 01:09:16.515956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:16.515856    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:16.910114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:16.910030    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr" event={"ID":"c4587c86-fe98-40d7-8473-30f6fb0f79dd","Type":"ContainerStarted","Data":"25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b"}
Oct 30 01:09:16.910114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:16.910080    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr" event={"ID":"c4587c86-fe98-40d7-8473-30f6fb0f79dd","Type":"ContainerStarted","Data":"3ad4dd065362df4aeac8e69e007e4c915f9005b25695dcbd29bdf7fc8a8254ee"}
Oct 30 01:09:16.953381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:16.953328    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr" podStartSLOduration=1.953307679 podStartE2EDuration="1.953307679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:16.950975476 +0000 UTC m=+1837.022299310" watchObservedRunningTime="2026-10-30 01:09:16.953307679 +0000 UTC m=+1837.024631514"
Oct 30 01:09:17.369216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.369182    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"]
Oct 30 01:09:17.369477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.369411    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76" podUID="faa20c39-a9fd-4b88-a6fa-c8211884757a" containerName="discovery" containerID="cri-o://3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c" gracePeriod=30
Oct 30 01:09:17.666135 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.666105    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:09:17.801528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.801496    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-token\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-token\") pod \"faa20c39-a9fd-4b88-a6fa-c8211884757a\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") "
Oct 30 01:09:17.801740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.801607    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-dns-cert\") pod \"faa20c39-a9fd-4b88-a6fa-c8211884757a\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") "
Oct 30 01:09:17.801740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.801645    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa20c39-a9fd-4b88-a6fa-c8211884757a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa20c39-a9fd-4b88-a6fa-c8211884757a-local-certs\") pod \"faa20c39-a9fd-4b88-a6fa-c8211884757a\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") "
Oct 30 01:09:17.801740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.801670    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-cacerts\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-cacerts\") pod \"faa20c39-a9fd-4b88-a6fa-c8211884757a\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") "
Oct 30 01:09:17.801901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.801769    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-kubeconfig\") pod \"faa20c39-a9fd-4b88-a6fa-c8211884757a\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") "
Oct 30 01:09:17.801901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.801815    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-kube-api-access-5h4zz\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-kube-api-access-5h4zz\") pod \"faa20c39-a9fd-4b88-a6fa-c8211884757a\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") "
Oct 30 01:09:17.801901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.801859    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-ca-configmap\") pod \"faa20c39-a9fd-4b88-a6fa-c8211884757a\" (UID: \"faa20c39-a9fd-4b88-a6fa-c8211884757a\") "
Oct 30 01:09:17.802615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.802566    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-ca-configmap" pod "faa20c39-a9fd-4b88-a6fa-c8211884757a" (UID: "faa20c39-a9fd-4b88-a6fa-c8211884757a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:17.804517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.804491    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-kubeconfig" pod "faa20c39-a9fd-4b88-a6fa-c8211884757a" (UID: "faa20c39-a9fd-4b88-a6fa-c8211884757a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:17.804703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.804669    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-cacerts" pod "faa20c39-a9fd-4b88-a6fa-c8211884757a" (UID: "faa20c39-a9fd-4b88-a6fa-c8211884757a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:17.804703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.804691    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa20c39-a9fd-4b88-a6fa-c8211884757a-local-certs" pod "faa20c39-a9fd-4b88-a6fa-c8211884757a" (UID: "faa20c39-a9fd-4b88-a6fa-c8211884757a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:17.804703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.804679    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-dns-cert" pod "faa20c39-a9fd-4b88-a6fa-c8211884757a" (UID: "faa20c39-a9fd-4b88-a6fa-c8211884757a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:17.804945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.804922    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-kube-api-access-5h4zz" pod "faa20c39-a9fd-4b88-a6fa-c8211884757a" (UID: "faa20c39-a9fd-4b88-a6fa-c8211884757a"). InnerVolumeSpecName "kube-api-access-5h4zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:17.805019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.805000    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-token" pod "faa20c39-a9fd-4b88-a6fa-c8211884757a" (UID: "faa20c39-a9fd-4b88-a6fa-c8211884757a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:17.903957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.903868    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h4zz\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-kube-api-access-5h4zz\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:17.903957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.903900    4576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-ca-configmap\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:17.903957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.903916    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:17.903957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.903928    4576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-csr-dns-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:17.903957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.903939    4576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa20c39-a9fd-4b88-a6fa-c8211884757a-local-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:17.903957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.903950    4576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-cacerts\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:17.903957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.903962    4576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/faa20c39-a9fd-4b88-a6fa-c8211884757a-istio-kubeconfig\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:17.907299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.907272    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:17.920033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.919995    4576 generic.go:356] "Generic (PLEG): container finished" podID="faa20c39-a9fd-4b88-a6fa-c8211884757a" containerID="3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c" exitCode=0
Oct 30 01:09:17.922224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.921466    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"
Oct 30 01:09:17.922224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.921710    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76" event={"ID":"faa20c39-a9fd-4b88-a6fa-c8211884757a","Type":"ContainerDied","Data":"3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c"}
Oct 30 01:09:17.922224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.921790    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76" event={"ID":"faa20c39-a9fd-4b88-a6fa-c8211884757a","Type":"ContainerDied","Data":"6422ee3606b53f628a8626b4b1ba011eab16e667c30c2740a4afe123e3f18412"}
Oct 30 01:09:17.922224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.921829    4576 scope.go:122] "RemoveContainer" containerID="3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c"
Oct 30 01:09:17.943678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.943658    4576 scope.go:122] "RemoveContainer" containerID="3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c"
Oct 30 01:09:17.946734 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:09:17.946711    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c\": container with ID starting with 3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c not found: ID does not exist" containerID="3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c"
Oct 30 01:09:17.946842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.946745    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c"} err="failed to get container status \"3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c\": rpc error: code = NotFound desc = could not find container \"3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c\": container with ID starting with 3c8ae06188210e4184a7b3db44617652c6c2dac08fa2ae5da1096202bebd199c not found: ID does not exist"
Oct 30 01:09:17.980032 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.979992    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"]
Oct 30 01:09:17.993444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:17.993421    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5t76"]
Oct 30 01:09:18.111858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:18.111821    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa20c39-a9fd-4b88-a6fa-c8211884757a" path="/var/lib/kubelet/pods/faa20c39-a9fd-4b88-a6fa-c8211884757a/volumes"
Oct 30 01:09:18.184823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:18.184732    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:18.384513 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:09:18.384465    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:09:19.748145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.748113    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"]
Oct 30 01:09:19.748789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.748773    4576 cpu_manager.go:414] "containerMap: removing container" podUID="faa20c39-a9fd-4b88-a6fa-c8211884757a" containerName="discovery"
Oct 30 01:09:19.748789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.748790    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa20c39-a9fd-4b88-a6fa-c8211884757a" containerName="discovery"
Oct 30 01:09:19.748890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.748881    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa20c39-a9fd-4b88-a6fa-c8211884757a" containerName="discovery"
Oct 30 01:09:19.753439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.753415    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.769754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.769726    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0680fa\"/\"kube-root-ca.crt\""
Oct 30 01:09:19.770417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.770393    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"]
Oct 30 01:09:19.780071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.780043    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0680fa\"/\"default-dockercfg-rhmv7\""
Oct 30 01:09:19.780222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.780057    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0680fa\"/\"openshift-service-ca.crt\""
Oct 30 01:09:19.780222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.780163    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0680fa\"/\"test-model-0680fa-kserve-self-signed-certs\""
Oct 30 01:09:19.828874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.828837    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndvnp\" (UniqueName: \"kubernetes.io/projected/2678af68-73b3-4ec2-a4e4-03e42c73b45b-kube-api-access-ndvnp\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.829062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.828900    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tmp-dir\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.829062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.829016    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tls-certs\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.829062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.829052    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-model-cache\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.829254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.829086    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-home\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.829424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.829384    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-dshm\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.846101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.846066    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:19.930295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.930246    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-dshm\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.930456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.930341    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndvnp\" (UniqueName: \"kubernetes.io/projected/2678af68-73b3-4ec2-a4e4-03e42c73b45b-kube-api-access-ndvnp\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.930456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.930402    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tmp-dir\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.930593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.930465    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tls-certs\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.930593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.930497    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-model-cache\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.930593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.930541    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-home\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.931026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.931005    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-model-cache\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.931190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.931005    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tmp-dir\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.931307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.931281    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-home\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.932942 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.932915    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-dshm\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.933910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.933889    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tls-certs\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.951519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.951488    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndvnp\" (UniqueName: \"kubernetes.io/projected/2678af68-73b3-4ec2-a4e4-03e42c73b45b-kube-api-access-ndvnp\") pod \"test-model-0680fa-kserve-c67cdc59c-6gb44\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") " pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:19.973384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:19.973356    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:20.078307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:20.077297    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:20.447584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:20.446960    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"]
Oct 30 01:09:20.447821 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:20.447794    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2678af68_73b3_4ec2_a4e4_03e42c73b45b.slice/crio-7f73f7c87c130fac82d5990d4a79bdcccbad835b83fe966b06261e159ebf6382 WatchSource:0}: Error finding container 7f73f7c87c130fac82d5990d4a79bdcccbad835b83fe966b06261e159ebf6382: Status 404 returned error can't find the container with id 7f73f7c87c130fac82d5990d4a79bdcccbad835b83fe966b06261e159ebf6382
Oct 30 01:09:20.949988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:20.949782    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44" event={"ID":"2678af68-73b3-4ec2-a4e4-03e42c73b45b","Type":"ContainerStarted","Data":"180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903"}
Oct 30 01:09:20.949988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:20.949827    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44" event={"ID":"2678af68-73b3-4ec2-a4e4-03e42c73b45b","Type":"ContainerStarted","Data":"7f73f7c87c130fac82d5990d4a79bdcccbad835b83fe966b06261e159ebf6382"}
Oct 30 01:09:20.995147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:20.995096    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44" podStartSLOduration=1.9950824630000001 podStartE2EDuration="1.995082463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:20.992529221 +0000 UTC m=+1841.063853057" watchObservedRunningTime="2026-10-30 01:09:20.995082463 +0000 UTC m=+1841.066406298"
Oct 30 01:09:21.077399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:21.077360    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:21.649476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:21.649443    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:23.308631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.308582    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:23.492880 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.492843    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"]
Oct 30 01:09:23.497474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.497446    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.519248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.519203    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"]
Oct 30 01:09:23.524629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.524603    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d54608\"/\"kube-root-ca.crt\""
Oct 30 01:09:23.524809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.524689    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d54608\"/\"openshift-service-ca.crt\""
Oct 30 01:09:23.534608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.534581    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d54608\"/\"test-model-d54608-kserve-self-signed-certs\""
Oct 30 01:09:23.534764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.534649    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d54608\"/\"default-dockercfg-nklzs\""
Oct 30 01:09:23.626124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.626006    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:23.686043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.686002    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvql8\" (UniqueName: \"kubernetes.io/projected/32f7439a-4727-4f66-99e6-716ca8003fc1-kube-api-access-pvql8\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.686250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.686056    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7439a-4727-4f66-99e6-716ca8003fc1-tls-certs\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.686250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.686146    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-dshm\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.686351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.686311    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-tmp-dir\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.686529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.686510    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-home\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.686699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.686674    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-model-cache\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.789358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.789313    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-home\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.789558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.788810    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-home\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.789558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.789516    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-model-cache\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.789696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.789603    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvql8\" (UniqueName: \"kubernetes.io/projected/32f7439a-4727-4f66-99e6-716ca8003fc1-kube-api-access-pvql8\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.789696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.789640    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7439a-4727-4f66-99e6-716ca8003fc1-tls-certs\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.789696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.789683    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-dshm\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.789871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.789728    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-tmp-dir\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.792292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.790274    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-tmp-dir\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.792292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.790528    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-model-cache\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.795614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.794528    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7439a-4727-4f66-99e6-716ca8003fc1-tls-certs\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.803386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.799847    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-dshm\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.812085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.812058    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvql8\" (UniqueName: \"kubernetes.io/projected/32f7439a-4727-4f66-99e6-716ca8003fc1-kube-api-access-pvql8\") pod \"test-model-d54608-kserve-684448976f-d88sl\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") " pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:23.821851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:23.821829    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:24.168873 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:24.168840    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32f7439a_4727_4f66_99e6_716ca8003fc1.slice/crio-2e50c7f096dad8dd8c164dfff022d0656bfa339ede3f835ba052a304cd4d4228 WatchSource:0}: Error finding container 2e50c7f096dad8dd8c164dfff022d0656bfa339ede3f835ba052a304cd4d4228: Status 404 returned error can't find the container with id 2e50c7f096dad8dd8c164dfff022d0656bfa339ede3f835ba052a304cd4d4228
Oct 30 01:09:24.169514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:24.169413    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"]
Oct 30 01:09:25.019992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:25.019954    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl" event={"ID":"32f7439a-4727-4f66-99e6-716ca8003fc1","Type":"ContainerStarted","Data":"a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f"}
Oct 30 01:09:25.020538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:25.020004    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl" event={"ID":"32f7439a-4727-4f66-99e6-716ca8003fc1","Type":"ContainerStarted","Data":"2e50c7f096dad8dd8c164dfff022d0656bfa339ede3f835ba052a304cd4d4228"}
Oct 30 01:09:25.066030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:25.065973    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl" podStartSLOduration=2.065957761 podStartE2EDuration="2.065957761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:25.06167839 +0000 UTC m=+1845.133002242" watchObservedRunningTime="2026-10-30 01:09:25.065957761 +0000 UTC m=+1845.137281595"
Oct 30 01:09:25.412367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:25.412268    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:25.822027 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:25.821992    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:25.907997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:25.907969    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:25.921122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:25.921095    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:26.038245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:26.038200    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:09:27.073295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:27.073261    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:28.082337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.082296    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"]
Oct 30 01:09:28.086956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.086932    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.114813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.113604    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-286d36\"/\"openshift-service-ca.crt\""
Oct 30 01:09:28.114813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.113734    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-286d36\"/\"kube-root-ca.crt\""
Oct 30 01:09:28.129001 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.124886    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-286d36\"/\"test-model-286d36-kserve-self-signed-certs\""
Oct 30 01:09:28.129001 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.125561    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-286d36\"/\"default-dockercfg-m9d2k\""
Oct 30 01:09:28.129001 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.127315    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"]
Oct 30 01:09:28.162272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.162218    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:28.167504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.167473    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hfsl\" (UniqueName: \"kubernetes.io/projected/8dcc55f9-46b7-4500-8891-fd949f2a972d-kube-api-access-7hfsl\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.167663 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.167606    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-dshm\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.167739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.167709    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcc55f9-46b7-4500-8891-fd949f2a972d-tls-certs\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.167797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.167776    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-home\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.167880 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.167864    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-model-cache\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.168128 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.168102    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-tmp-dir\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.269603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.269562    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcc55f9-46b7-4500-8891-fd949f2a972d-tls-certs\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.269829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.269637    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-home\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.269829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.269680    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-model-cache\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.269947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.269853    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-tmp-dir\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.269998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.269952    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hfsl\" (UniqueName: \"kubernetes.io/projected/8dcc55f9-46b7-4500-8891-fd949f2a972d-kube-api-access-7hfsl\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.270053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.270019    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-dshm\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.270123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.270090    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-home\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.270182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.270147    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-model-cache\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.270443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.270420    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-tmp-dir\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.272371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.272351    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-dshm\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.272775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.272760    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcc55f9-46b7-4500-8891-fd949f2a972d-tls-certs\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.291334 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.291301    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hfsl\" (UniqueName: \"kubernetes.io/projected/8dcc55f9-46b7-4500-8891-fd949f2a972d-kube-api-access-7hfsl\") pod \"test-model-286d36-kserve-8584c47669-cdxd9\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") " pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.409407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.409313    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:28.756551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.756521    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"]
Oct 30 01:09:28.757650 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:28.757623    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dcc55f9_46b7_4500_8891_fd949f2a972d.slice/crio-f5eb35c219ceddd76dedede29b7429326831386022ca187eb930d583fb52b16c WatchSource:0}: Error finding container f5eb35c219ceddd76dedede29b7429326831386022ca187eb930d583fb52b16c: Status 404 returned error can't find the container with id f5eb35c219ceddd76dedede29b7429326831386022ca187eb930d583fb52b16c
Oct 30 01:09:28.925124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:28.925088    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:29.049935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:29.049846    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9" event={"ID":"8dcc55f9-46b7-4500-8891-fd949f2a972d","Type":"ContainerStarted","Data":"07a14540cf63922aa28c9ec8a27babd68d9859bd93d0cd338c7148f5a1fdba53"}
Oct 30 01:09:29.049935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:29.049895    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9" event={"ID":"8dcc55f9-46b7-4500-8891-fd949f2a972d","Type":"ContainerStarted","Data":"f5eb35c219ceddd76dedede29b7429326831386022ca187eb930d583fb52b16c"}
Oct 30 01:09:29.094068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:29.094007    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9" podStartSLOduration=1.093992546 podStartE2EDuration="1.093992546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:29.089993328 +0000 UTC m=+1849.161317162" watchObservedRunningTime="2026-10-30 01:09:29.093992546 +0000 UTC m=+1849.165316382"
Oct 30 01:09:29.409742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:29.409664    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:30.078453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:30.078412    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:30.092198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:30.092172    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:30.278968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:30.278927    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:30.966547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:30.966515    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:31.078044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:31.078010    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:09:31.247273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:31.247164    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:31.303424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:31.303394    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"]
Oct 30 01:09:31.308888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:31.308854    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bd9cc5-zpd8g"]
Oct 30 01:09:31.408540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:31.408508    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:31.767060 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:31.767028    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:32.109794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:32.109715    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1a3c90-2d93-4d37-8125-bda4e7cc2c81" path="/var/lib/kubelet/pods/0a1a3c90-2d93-4d37-8125-bda4e7cc2c81/volumes"
Oct 30 01:09:32.846711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:32.846676    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:32.953297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:32.953266    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:33.239435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.239402    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:33.428526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.428495    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"]
Oct 30 01:09:33.432984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.432967    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.451745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.451706    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"]
Oct 30 01:09:33.460372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.460340    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:33.460372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.460348    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:09:33.553557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.553475    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5km2x\" (UniqueName: \"kubernetes.io/projected/88e57499-5390-46c8-9989-c866926286c4-kube-api-access-5km2x\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.553722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.553679    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e57499-5390-46c8-9989-c866926286c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.553801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.553786    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e57499-5390-46c8-9989-c866926286c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.654668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.654629    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5km2x\" (UniqueName: \"kubernetes.io/projected/88e57499-5390-46c8-9989-c866926286c4-kube-api-access-5km2x\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.654856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.654813    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e57499-5390-46c8-9989-c866926286c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.654953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.654936    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e57499-5390-46c8-9989-c866926286c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.655631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.655610    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e57499-5390-46c8-9989-c866926286c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.657446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.657416    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e57499-5390-46c8-9989-c866926286c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.675688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.675655    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5km2x\" (UniqueName: \"kubernetes.io/projected/88e57499-5390-46c8-9989-c866926286c4-kube-api-access-5km2x\") pod \"maas-api-revoke-sub-13b5d43a-brbsn\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.748701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.748666    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:33.825250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.823787    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:33.844843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:33.844817    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:34.051208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:34.051177    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"]
Oct 30 01:09:34.052651 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:34.052621    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e57499_5390_46c8_9989_c866926286c4.slice/crio-8c6d0985aa32c838e89d78339e19a4e6c2794102a90d8293aa742b7bf8a77324 WatchSource:0}: Error finding container 8c6d0985aa32c838e89d78339e19a4e6c2794102a90d8293aa742b7bf8a77324: Status 404 returned error can't find the container with id 8c6d0985aa32c838e89d78339e19a4e6c2794102a90d8293aa742b7bf8a77324
Oct 30 01:09:34.085941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:34.085907    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn" event={"ID":"88e57499-5390-46c8-9989-c866926286c4","Type":"ContainerStarted","Data":"8c6d0985aa32c838e89d78339e19a4e6c2794102a90d8293aa742b7bf8a77324"}
Oct 30 01:09:34.098209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:34.098186    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:09:34.723560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:34.723518    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:35.017268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:35.017210    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:35.094638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:35.094603    4576 generic.go:356] "Generic (PLEG): container finished" podID="88e57499-5390-46c8-9989-c866926286c4" containerID="06bc3fd66e8d8034b6cce4926bbd99ab4ba65837bd1bf174c0589ff3cf2e5b62" exitCode=0
Oct 30 01:09:35.094638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:35.094640    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn" event={"ID":"88e57499-5390-46c8-9989-c866926286c4","Type":"ContainerDied","Data":"06bc3fd66e8d8034b6cce4926bbd99ab4ba65837bd1bf174c0589ff3cf2e5b62"}
Oct 30 01:09:36.210425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.210388    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:36.273460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.273432    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:36.399382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.399298    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e57499-5390-46c8-9989-c866926286c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e57499-5390-46c8-9989-c866926286c4-maas-api-xks-ca\") pod \"88e57499-5390-46c8-9989-c866926286c4\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") "
Oct 30 01:09:36.399382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.399349    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e57499-5390-46c8-9989-c866926286c4-kube-api-access-5km2x\" (UniqueName: \"kubernetes.io/projected/88e57499-5390-46c8-9989-c866926286c4-kube-api-access-5km2x\") pod \"88e57499-5390-46c8-9989-c866926286c4\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") "
Oct 30 01:09:36.399573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.399408    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e57499-5390-46c8-9989-c866926286c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e57499-5390-46c8-9989-c866926286c4-maas-api-openshift-ca\") pod \"88e57499-5390-46c8-9989-c866926286c4\" (UID: \"88e57499-5390-46c8-9989-c866926286c4\") "
Oct 30 01:09:36.399870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.399837    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e57499-5390-46c8-9989-c866926286c4-maas-api-openshift-ca" pod "88e57499-5390-46c8-9989-c866926286c4" (UID: "88e57499-5390-46c8-9989-c866926286c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:36.400090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.400074    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e57499-5390-46c8-9989-c866926286c4-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:36.401697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.401669    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e57499-5390-46c8-9989-c866926286c4-maas-api-xks-ca" pod "88e57499-5390-46c8-9989-c866926286c4" (UID: "88e57499-5390-46c8-9989-c866926286c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:36.401847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.401821    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e57499-5390-46c8-9989-c866926286c4-kube-api-access-5km2x" pod "88e57499-5390-46c8-9989-c866926286c4" (UID: "88e57499-5390-46c8-9989-c866926286c4"). InnerVolumeSpecName "kube-api-access-5km2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:36.502010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.501980    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e57499-5390-46c8-9989-c866926286c4-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:36.502010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.502009    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5km2x\" (UniqueName: \"kubernetes.io/projected/88e57499-5390-46c8-9989-c866926286c4-kube-api-access-5km2x\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:36.640178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:36.640142    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:37.110935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:37.110904    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"
Oct 30 01:09:37.111223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:37.110933    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn" event={"ID":"88e57499-5390-46c8-9989-c866926286c4","Type":"ContainerDied","Data":"8c6d0985aa32c838e89d78339e19a4e6c2794102a90d8293aa742b7bf8a77324"}
Oct 30 01:09:37.111405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:37.111278    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c6d0985aa32c838e89d78339e19a4e6c2794102a90d8293aa742b7bf8a77324"
Oct 30 01:09:37.873385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:37.873350    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:38.250961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:38.250929    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:38.365469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:38.365420    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:38.409775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:38.409744    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:38.422679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:38.422652    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:38.734085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:38.734048    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:39.116518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.116434    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:39.136122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.136095    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:09:39.401098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.401006    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:39.566624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.566581    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:39.785114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.785079    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"]
Oct 30 01:09:39.786088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.786063    4576 cpu_manager.go:414] "containerMap: removing container" podUID="88e57499-5390-46c8-9989-c866926286c4" containerName="revoke-keys"
Oct 30 01:09:39.786088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.786087    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e57499-5390-46c8-9989-c866926286c4" containerName="revoke-keys"
Oct 30 01:09:39.786331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.786311    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e57499-5390-46c8-9989-c866926286c4" containerName="revoke-keys"
Oct 30 01:09:39.790705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.790683    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.814919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.811628    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"]
Oct 30 01:09:39.820067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.819468    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b312f7-openshift-default-dockercfg-t8sp7\""
Oct 30 01:09:39.867007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.866970    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:39.869657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.869622    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-credential-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.869823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.869689    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-certs\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.869823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.869803    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-token\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.869952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.869856    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz8lm\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-kube-api-access-jz8lm\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.869952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.869920    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-data\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.870071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.870042    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-envoy\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.870203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.870122    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-podinfo\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.870370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.870350    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028e239-916a-4474-9da9-bb2f47dfb84d-istiod-ca-cert\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.870474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.870386    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.971566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.971533    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028e239-916a-4474-9da9-bb2f47dfb84d-istiod-ca-cert\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.971566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.971577    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.971862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.971677    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-credential-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.971862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.971723    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-certs\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.971862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.971768    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-token\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.971862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.971808    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz8lm\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-kube-api-access-jz8lm\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.972061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.971864    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-data\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.972061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.971987    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-envoy\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.972061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.972015    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-podinfo\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.972632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.972481    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028e239-916a-4474-9da9-bb2f47dfb84d-istiod-ca-cert\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.973164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.973142    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.973453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.973432    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-credential-socket\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.973690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.973672    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-certs\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.973920 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.973899    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-data\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:39.976891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:39.976870    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-envoy\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:40.006261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.004905    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-podinfo\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:40.017259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.015137    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz8lm\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-kube-api-access-jz8lm\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:40.036419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.032037    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-token\") pod \"e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") " pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:40.144763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.144730    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b312f7-openshift-default-dockercfg-t8sp7\""
Oct 30 01:09:40.145991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.145957    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:40.173795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.164997    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:40.541550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.541518    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"]
Oct 30 01:09:40.543355 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:40.543334    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4028e239_916a_4474_9da9_bb2f47dfb84d.slice/crio-b39d0966e54ad7e69124ee8fe95a8e55fb364b1f8d025079038f468c324f776a WatchSource:0}: Error finding container b39d0966e54ad7e69124ee8fe95a8e55fb364b1f8d025079038f468c324f776a: Status 404 returned error can't find the container with id b39d0966e54ad7e69124ee8fe95a8e55fb364b1f8d025079038f468c324f776a
Oct 30 01:09:40.546488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.546456    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:09:40.546567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.546528    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:09:40.546567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:40.546560    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:09:41.177964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:41.177913    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" event={"ID":"4028e239-916a-4474-9da9-bb2f47dfb84d","Type":"ContainerStarted","Data":"e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c"}
Oct 30 01:09:41.178434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:41.178049    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" event={"ID":"4028e239-916a-4474-9da9-bb2f47dfb84d","Type":"ContainerStarted","Data":"b39d0966e54ad7e69124ee8fe95a8e55fb364b1f8d025079038f468c324f776a"}
Oct 30 01:09:41.222609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:41.222544    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" podStartSLOduration=2.222526384 podStartE2EDuration="2.222526384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:41.222212568 +0000 UTC m=+1861.293536439" watchObservedRunningTime="2026-10-30 01:09:41.222526384 +0000 UTC m=+1861.293850220"
Oct 30 01:09:41.332748 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:41.331403    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:42.156295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:42.156261    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:42.162898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:42.162867    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:42.185402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:42.185363    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:42.186282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:42.186266    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:09:43.077073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.077031    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:43.207603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.207565    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"]
Oct 30 01:09:43.215399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.215370    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.236968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.236935    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"]
Oct 30 01:09:43.243447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.243417    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bc6092\"/\"openshift-service-ca.crt\""
Oct 30 01:09:43.243668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.243473    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bc6092\"/\"default-dockercfg-5c2jl\""
Oct 30 01:09:43.246753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.246731    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bc6092\"/\"kube-root-ca.crt\""
Oct 30 01:09:43.246908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.246774    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bc6092\"/\"test-model-bc6092-kserve-self-signed-certs\""
Oct 30 01:09:43.279569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.279536    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:43.353628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.353544    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-model-cache\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.353628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.353584    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-dshm\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.353842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.353709    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tls-certs\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.353842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.353830    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg962\" (UniqueName: \"kubernetes.io/projected/9a1c927d-c2cc-44ef-a2b9-323e0866a101-kube-api-access-wg962\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.353952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.353906    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tmp-dir\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.354225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.354198    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-home\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.458295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.455565    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-model-cache\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.458295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.455610    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-dshm\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.458295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.455672    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tls-certs\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.458295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.455712    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg962\" (UniqueName: \"kubernetes.io/projected/9a1c927d-c2cc-44ef-a2b9-323e0866a101-kube-api-access-wg962\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.458295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.455762    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tmp-dir\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.458295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.456007    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-home\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.458295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.456474    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-home\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.458295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.456698    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-model-cache\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.462259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.459532    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-dshm\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.474291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.473264    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tmp-dir\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.480665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.480596    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tls-certs\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.506125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.506091    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg962\" (UniqueName: \"kubernetes.io/projected/9a1c927d-c2cc-44ef-a2b9-323e0866a101-kube-api-access-wg962\") pod \"test-model-bc6092-kserve-7d7b64f75f-nhcq9\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") " pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.537988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.537935    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:43.741289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.741250    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:43.870393 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:43.870356    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a1c927d_c2cc_44ef_a2b9_323e0866a101.slice/crio-549441a8e232f8a47f3ee0c3e1f13859840232017734226c608f6876b6ec6f6e WatchSource:0}: Error finding container 549441a8e232f8a47f3ee0c3e1f13859840232017734226c608f6876b6ec6f6e: Status 404 returned error can't find the container with id 549441a8e232f8a47f3ee0c3e1f13859840232017734226c608f6876b6ec6f6e
Oct 30 01:09:43.870694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.870677    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"]
Oct 30 01:09:43.871791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:43.871765    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:44.202409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:44.202370    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9" event={"ID":"9a1c927d-c2cc-44ef-a2b9-323e0866a101","Type":"ContainerStarted","Data":"4c84b082948a5f609ef5cafcb2ade85bdc60a20e9fe856c67918b780f3161d3f"}
Oct 30 01:09:44.202601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:44.202422    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9" event={"ID":"9a1c927d-c2cc-44ef-a2b9-323e0866a101","Type":"ContainerStarted","Data":"549441a8e232f8a47f3ee0c3e1f13859840232017734226c608f6876b6ec6f6e"}
Oct 30 01:09:44.248912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:44.248855    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9" podStartSLOduration=1.248840333 podStartE2EDuration="1.248840333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:44.24552186 +0000 UTC m=+1864.316845695" watchObservedRunningTime="2026-10-30 01:09:44.248840333 +0000 UTC m=+1864.320164167"
Oct 30 01:09:44.540848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:44.540766    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:44.984225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:44.984184    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:45.639698 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.639662    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:45.771663 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.771625    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:45.790400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.790364    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"]
Oct 30 01:09:45.794659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.794637    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:45.811777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.811748    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"]
Oct 30 01:09:45.820988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.820963    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b312f7-dockercfg-jz8c7\""
Oct 30 01:09:45.878556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.878524    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b312f7\""
Oct 30 01:09:45.894751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.894648    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6576139-2679-4a30-b6dd-ec97d4dd6a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l\" (UID: \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:45.895196 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:45.895172    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5g8\" (UniqueName: \"kubernetes.io/projected/e6576139-2679-4a30-b6dd-ec97d4dd6a41-kube-api-access-gk5g8\") pod \"payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l\" (UID: \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:46.009303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:46.002629    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5g8\" (UniqueName: \"kubernetes.io/projected/e6576139-2679-4a30-b6dd-ec97d4dd6a41-kube-api-access-gk5g8\") pod \"payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l\" (UID: \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:46.009303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:46.002893    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6576139-2679-4a30-b6dd-ec97d4dd6a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l\" (UID: \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:46.009303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:46.003951    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6576139-2679-4a30-b6dd-ec97d4dd6a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l\" (UID: \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:46.026387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:46.026343    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5g8\" (UniqueName: \"kubernetes.io/projected/e6576139-2679-4a30-b6dd-ec97d4dd6a41-kube-api-access-gk5g8\") pod \"payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l\" (UID: \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:46.111718 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:46.111687    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:46.424372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:46.424307    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"]
Oct 30 01:09:46.424915 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:46.424894    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6576139_2679_4a30_b6dd_ec97d4dd6a41.slice/crio-1eeff144b5196194a5e219b7b04531a305a0834ac0ac2b765e674ce37939e1c3 WatchSource:0}: Error finding container 1eeff144b5196194a5e219b7b04531a305a0834ac0ac2b765e674ce37939e1c3: Status 404 returned error can't find the container with id 1eeff144b5196194a5e219b7b04531a305a0834ac0ac2b765e674ce37939e1c3
Oct 30 01:09:46.703459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:46.703425    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:47.225608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:47.225578    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l" event={"ID":"e6576139-2679-4a30-b6dd-ec97d4dd6a41","Type":"ContainerStarted","Data":"aba30fa127335f5f4dd65447711ffb1d151024d65a5d170c06a2ac6328440061"}
Oct 30 01:09:47.225816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:47.225625    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l" event={"ID":"e6576139-2679-4a30-b6dd-ec97d4dd6a41","Type":"ContainerStarted","Data":"1eeff144b5196194a5e219b7b04531a305a0834ac0ac2b765e674ce37939e1c3"}
Oct 30 01:09:47.225816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:47.225726    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:48.383291 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:09:48.383210    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:09:48.416026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:48.415975    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:49.576888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.576824    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l" podStartSLOduration=4.576803566 podStartE2EDuration="4.576803566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:47.268031166 +0000 UTC m=+1867.339355003" watchObservedRunningTime="2026-10-30 01:09:49.576803566 +0000 UTC m=+1869.648127404"
Oct 30 01:09:49.579109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.579084    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"]
Oct 30 01:09:49.584623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.584601    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.600989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.600959    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"]
Oct 30 01:09:49.613529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.613503    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-42d56b\"/\"openshift-service-ca.crt\""
Oct 30 01:09:49.613704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.613511    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-42d56b\"/\"kube-root-ca.crt\""
Oct 30 01:09:49.613704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.613656    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-42d56b\"/\"default-dockercfg-k6k8h\""
Oct 30 01:09:49.615611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.615584    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-42d56b\"/\"auth-test-model-42d56b-kserve-self-signed-certs\""
Oct 30 01:09:49.647438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.647360    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:49.756382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.756352    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-dshm\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.756567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.756482    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-home\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.756567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.756530    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tmp-dir\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.756652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.756595    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-model-cache\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.756652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.756616    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tls-certs\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.756765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.756751    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmv4\" (UniqueName: \"kubernetes.io/projected/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-kube-api-access-7mmv4\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.858664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.858569    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmv4\" (UniqueName: \"kubernetes.io/projected/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-kube-api-access-7mmv4\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.858823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.858660    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-dshm\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.858823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.858789    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-home\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.858823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.858815    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tmp-dir\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.858998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.858859    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-model-cache\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.858998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.858884    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tls-certs\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.859209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.859184    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-home\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.859302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.859269    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tmp-dir\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.859506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.859483    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-model-cache\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.861291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.861227    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-dshm\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.861713 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.861692    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tls-certs\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.881141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.881103    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmv4\" (UniqueName: \"kubernetes.io/projected/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-kube-api-access-7mmv4\") pod \"auth-test-model-42d56b-kserve-75c689f87c-s9xwk\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:49.908090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:49.908054    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:50.058001 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:50.057833    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:50.264189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:50.264158    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"]
Oct 30 01:09:50.264668 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:50.264643    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ab39d4c_b152_4fe4_93f1_4411e5fdddd1.slice/crio-8001f35948bc2ebaddbc3171399c2fab2dce0e6fee0e0db45b0e04c47f9b9a5d WatchSource:0}: Error finding container 8001f35948bc2ebaddbc3171399c2fab2dce0e6fee0e0db45b0e04c47f9b9a5d: Status 404 returned error can't find the container with id 8001f35948bc2ebaddbc3171399c2fab2dce0e6fee0e0db45b0e04c47f9b9a5d
Oct 30 01:09:51.257390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:51.257351    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk" event={"ID":"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1","Type":"ContainerStarted","Data":"f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0"}
Oct 30 01:09:51.257797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:51.257407    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk" event={"ID":"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1","Type":"ContainerStarted","Data":"8001f35948bc2ebaddbc3171399c2fab2dce0e6fee0e0db45b0e04c47f9b9a5d"}
Oct 30 01:09:51.301926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:51.301875    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk" podStartSLOduration=2.301861318 podStartE2EDuration="2.301861318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:51.299433202 +0000 UTC m=+1871.370757047" watchObservedRunningTime="2026-10-30 01:09:51.301861318 +0000 UTC m=+1871.373185153"
Oct 30 01:09:51.685293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:51.685165    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:51.908184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:51.908156    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:53.372002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:53.371966    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:53.540985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:53.540950    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:53.554098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:53.554072    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:54.289342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:54.289317    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:09:55.227340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:55.210945    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:56.028795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.028757    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"]
Oct 30 01:09:56.033735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.033712    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.050338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.050306    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"]
Oct 30 01:09:56.059147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.059116    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:56.059147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.059141    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:09:56.152646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.152599    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.152924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.152902    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.153029 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.152945    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88rt\" (UniqueName: \"kubernetes.io/projected/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-kube-api-access-k88rt\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.262256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.259348    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.262256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.261517    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:56.262872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.262498    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.262872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.262541    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88rt\" (UniqueName: \"kubernetes.io/projected/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-kube-api-access-k88rt\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.281976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.273493    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.295257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.291718    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.295257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.294912    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88rt\" (UniqueName: \"kubernetes.io/projected/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-kube-api-access-k88rt\") pod \"maas-api-revoke-sub-8e789959-j4xw8\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.355694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.354468    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:56.734509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:56.733871    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"]
Oct 30 01:09:56.736189 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:09:56.736157    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a2f886_c8aa_455a_af8e_f0a35fae0c8c.slice/crio-e7648a2a2e0e5277452239e3650a51789f9595dcd81bf86d1c58f27801d53735 WatchSource:0}: Error finding container e7648a2a2e0e5277452239e3650a51789f9595dcd81bf86d1c58f27801d53735: Status 404 returned error can't find the container with id e7648a2a2e0e5277452239e3650a51789f9595dcd81bf86d1c58f27801d53735
Oct 30 01:09:57.435507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:57.435417    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8" event={"ID":"97a2f886-c8aa-455a-af8e-f0a35fae0c8c","Type":"ContainerStarted","Data":"625643df1db62418f62b5624cb98570f5786360505c88c9d056ba6d47519971b"}
Oct 30 01:09:57.435507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:57.435476    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8" event={"ID":"97a2f886-c8aa-455a-af8e-f0a35fae0c8c","Type":"ContainerStarted","Data":"e7648a2a2e0e5277452239e3650a51789f9595dcd81bf86d1c58f27801d53735"}
Oct 30 01:09:57.477442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:57.477381    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8" podStartSLOduration=1.477366753 podStartE2EDuration="1.477366753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:57.476072211 +0000 UTC m=+1877.547396048" watchObservedRunningTime="2026-10-30 01:09:57.477366753 +0000 UTC m=+1877.548690588"
Oct 30 01:09:58.074466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:58.074377    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:58.236568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:58.236543    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:09:58.444848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:58.444814    4576 generic.go:356] "Generic (PLEG): container finished" podID="97a2f886-c8aa-455a-af8e-f0a35fae0c8c" containerID="625643df1db62418f62b5624cb98570f5786360505c88c9d056ba6d47519971b" exitCode=0
Oct 30 01:09:58.445340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:58.444902    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8" event={"ID":"97a2f886-c8aa-455a-af8e-f0a35fae0c8c","Type":"ContainerDied","Data":"625643df1db62418f62b5624cb98570f5786360505c88c9d056ba6d47519971b"}
Oct 30 01:09:59.626353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.626328    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:09:59.724578 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.724501    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-kube-api-access-k88rt\" (UniqueName: \"kubernetes.io/projected/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-kube-api-access-k88rt\") pod \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") "
Oct 30 01:09:59.724578 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.724570    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-openshift-ca\") pod \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") "
Oct 30 01:09:59.724765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.724754    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-xks-ca\") pod \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\" (UID: \"97a2f886-c8aa-455a-af8e-f0a35fae0c8c\") "
Oct 30 01:09:59.725002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.724966    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-openshift-ca" pod "97a2f886-c8aa-455a-af8e-f0a35fae0c8c" (UID: "97a2f886-c8aa-455a-af8e-f0a35fae0c8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:59.725404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.725361    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.726939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.726913    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-xks-ca" pod "97a2f886-c8aa-455a-af8e-f0a35fae0c8c" (UID: "97a2f886-c8aa-455a-af8e-f0a35fae0c8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:59.727017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.726936    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-kube-api-access-k88rt" pod "97a2f886-c8aa-455a-af8e-f0a35fae0c8c" (UID: "97a2f886-c8aa-455a-af8e-f0a35fae0c8c"). InnerVolumeSpecName "kube-api-access-k88rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:59.744458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.744431    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:09:59.826889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.826853    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.826889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.826891    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88rt\" (UniqueName: \"kubernetes.io/projected/97a2f886-c8aa-455a-af8e-f0a35fae0c8c-kube-api-access-k88rt\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.908657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.908626    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:09:59.922007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:09:59.921981    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:10:00.465352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:00.465320    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"
Oct 30 01:10:00.465526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:00.465373    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8" event={"ID":"97a2f886-c8aa-455a-af8e-f0a35fae0c8c","Type":"ContainerDied","Data":"e7648a2a2e0e5277452239e3650a51789f9595dcd81bf86d1c58f27801d53735"}
Oct 30 01:10:00.465526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:00.465434    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7648a2a2e0e5277452239e3650a51789f9595dcd81bf86d1c58f27801d53735"
Oct 30 01:10:00.477608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:00.477581    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:10:01.280605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:01.280564    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:02.759838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:02.759799    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:04.459740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:04.459705    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:05.963361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:05.963327    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:07.558009 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:07.557964    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:09.270837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:09.270802    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:10.985499 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:10.985457    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:11.102132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:11.102085    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:12.422873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:12.422843    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:14.178341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:14.178309    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:15.722308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:15.722262    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:16.307752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.307715    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"]
Oct 30 01:10:16.308480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.308464    4576 cpu_manager.go:414] "containerMap: removing container" podUID="97a2f886-c8aa-455a-af8e-f0a35fae0c8c" containerName="revoke-keys"
Oct 30 01:10:16.308549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.308482    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a2f886-c8aa-455a-af8e-f0a35fae0c8c" containerName="revoke-keys"
Oct 30 01:10:16.308634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.308624    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a2f886-c8aa-455a-af8e-f0a35fae0c8c" containerName="revoke-keys"
Oct 30 01:10:16.314010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.313994    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.348736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.344489    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49d9f1-openshift-default-dockercfg-svv8m\""
Oct 30 01:10:16.349023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.348990    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"]
Oct 30 01:10:16.466938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.466900    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-envoy\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.467128 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.467036    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.467202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.467150    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istiod-ca-cert\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.467336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.467313    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-podinfo\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.467412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.467350    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-credential-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.467483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.467462    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2l7h\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-kube-api-access-t2l7h\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.467539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.467505    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-token\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.467588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.467536    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-certs\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.467632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.467618    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-data\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569031    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-podinfo\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569086    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-credential-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569157    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2l7h\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-kube-api-access-t2l7h\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569189    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-token\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569221    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-certs\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569272    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-data\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569323    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-envoy\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569425    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569534    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istiod-ca-cert\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569582    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-credential-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569709    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-certs\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.569961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.569938    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-socket\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.570183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.570162    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-data\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.570362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.570340    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istiod-ca-cert\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.572084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.572036    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-podinfo\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.572190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.572134    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-envoy\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.590218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.590192    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-token\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.590369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.590311    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2l7h\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-kube-api-access-t2l7h\") pod \"e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") " pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.628106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.628072    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:16.949818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.949792    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"]
Oct 30 01:10:16.950579 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:10:16.950554    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0822bea0_f14a_408f_8dd1_6dfa89de15b1.slice/crio-dde93ca2273a4a00fbfb69f159608447e8a349f54b366b8c9d1725dd1fee66ce WatchSource:0}: Error finding container dde93ca2273a4a00fbfb69f159608447e8a349f54b366b8c9d1725dd1fee66ce: Status 404 returned error can't find the container with id dde93ca2273a4a00fbfb69f159608447e8a349f54b366b8c9d1725dd1fee66ce
Oct 30 01:10:16.953423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.953386    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:16.953520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.953458    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:16.953520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:16.953486    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:17.360472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:17.360388    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:17.591171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:17.591122    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" event={"ID":"0822bea0-f14a-408f-8dd1-6dfa89de15b1","Type":"ContainerStarted","Data":"c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855"}
Oct 30 01:10:17.591351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:17.591187    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" event={"ID":"0822bea0-f14a-408f-8dd1-6dfa89de15b1","Type":"ContainerStarted","Data":"dde93ca2273a4a00fbfb69f159608447e8a349f54b366b8c9d1725dd1fee66ce"}
Oct 30 01:10:17.628438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:17.628366    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:17.636595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:17.636547    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" podStartSLOduration=1.636531398 podStartE2EDuration="1.636531398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:17.634281869 +0000 UTC m=+1897.705605708" watchObservedRunningTime="2026-10-30 01:10:17.636531398 +0000 UTC m=+1897.707855232"
Oct 30 01:10:18.408243 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:10:18.408184    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:10:18.630510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:18.630473    4576 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:10:18.630673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:18.630531    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:10:19.215258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.206838    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:19.320257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.319899    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:19.429431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.429387    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"]
Oct 30 01:10:19.440774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.440747    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.445839 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.445794    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"]
Oct 30 01:10:19.474654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.474555    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:10:19.482330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.482302    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:10:19.543626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.543570    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z77xv\" (UniqueName: \"kubernetes.io/projected/0b08a05f-6134-4a42-a611-2a14c7ec0628-kube-api-access-z77xv\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.544015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.543979    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.544365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.544345    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.628530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.628498    4576 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:10:19.628755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.628729    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:10:19.645431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.645396    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.645632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.645525    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z77xv\" (UniqueName: \"kubernetes.io/projected/0b08a05f-6134-4a42-a611-2a14c7ec0628-kube-api-access-z77xv\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.645632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.645586    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.646148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.646122    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.648126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.648094    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.704206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.704085    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z77xv\" (UniqueName: \"kubernetes.io/projected/0b08a05f-6134-4a42-a611-2a14c7ec0628-kube-api-access-z77xv\") pod \"maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:19.705778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.705752    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:10:19.855831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:19.855741    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:20.201404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.201367    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"]
Oct 30 01:10:20.562724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.562692    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"]
Oct 30 01:10:20.585296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.583668    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.602384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.602335    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"]
Oct 30 01:10:20.602738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.602716    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-286d36-openshift-default-dockercfg-qmrk9\""
Oct 30 01:10:20.625857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.625827    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b" event={"ID":"0b08a05f-6134-4a42-a611-2a14c7ec0628","Type":"ContainerStarted","Data":"d66f6488e4f597f931ce36237022c31be460158f9fbbc20d81fe5708674fbf05"}
Oct 30 01:10:20.626028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.625879    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b" event={"ID":"0b08a05f-6134-4a42-a611-2a14c7ec0628","Type":"ContainerStarted","Data":"0435c4e527e6ce88cf75fd99e9002347e7e29f7dc47003b1266d2a1a8309b140"}
Oct 30 01:10:20.629338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.629307    4576 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:10:20.629470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.629364    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:10:20.662133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662099    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhbz\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-kube-api-access-gkhbz\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.662351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662190    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-data\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.662351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662316    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-envoy\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.662469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662376    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d55436d-2557-4b59-8409-6739cd1b4551-istiod-ca-cert\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.662469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662438    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-certs\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.662469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662464    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d55436d-2557-4b59-8409-6739cd1b4551-istio-podinfo\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.662587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662565    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-credential-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.662638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662608    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.662784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.662763    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-istio-token\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.694484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.694425    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b" podStartSLOduration=1.69439147 podStartE2EDuration="1.69439147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:20.692536399 +0000 UTC m=+1900.763860236" watchObservedRunningTime="2026-10-30 01:10:20.69439147 +0000 UTC m=+1900.765715306"
Oct 30 01:10:20.764227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764193    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d55436d-2557-4b59-8409-6739cd1b4551-istiod-ca-cert\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764280    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-certs\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764318    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d55436d-2557-4b59-8409-6739cd1b4551-istio-podinfo\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764397    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-credential-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764446    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764495    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-istio-token\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764570    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhbz\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-kube-api-access-gkhbz\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764696    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-data\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764714    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-certs\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.764851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.764748    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-envoy\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.765021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.765000    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d55436d-2557-4b59-8409-6739cd1b4551-istiod-ca-cert\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.765388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.765362    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.765388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.765382    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-data\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.765531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.765396    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-credential-socket\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.767013 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.766992    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d55436d-2557-4b59-8409-6739cd1b4551-istio-podinfo\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.781262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.767296    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-envoy\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.798918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.798891    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhbz\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-kube-api-access-gkhbz\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.800129 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.800111    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-istio-token\") pod \"e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") " pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:20.919293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:20.919191    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:21.456726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.456689    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:21.463197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.463166    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"]
Oct 30 01:10:21.463614 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:10:21.463565    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d55436d_2557_4b59_8409_6739cd1b4551.slice/crio-742616ebcd4d4b72026b21a5452a72512fbeb192eda871678a072c193b51c2ae WatchSource:0}: Error finding container 742616ebcd4d4b72026b21a5452a72512fbeb192eda871678a072c193b51c2ae: Status 404 returned error can't find the container with id 742616ebcd4d4b72026b21a5452a72512fbeb192eda871678a072c193b51c2ae
Oct 30 01:10:21.466654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.466617    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:21.466736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.466691    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:21.466736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.466722    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:21.649285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.649220    4576 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:10:21.650073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.650023    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:10:21.744397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.741692    4576 generic.go:356] "Generic (PLEG): container finished" podID="0b08a05f-6134-4a42-a611-2a14c7ec0628" containerID="d66f6488e4f597f931ce36237022c31be460158f9fbbc20d81fe5708674fbf05" exitCode=0
Oct 30 01:10:21.744397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.741849    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b" event={"ID":"0b08a05f-6134-4a42-a611-2a14c7ec0628","Type":"ContainerDied","Data":"d66f6488e4f597f931ce36237022c31be460158f9fbbc20d81fe5708674fbf05"}
Oct 30 01:10:21.764264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.753736    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" event={"ID":"9d55436d-2557-4b59-8409-6739cd1b4551","Type":"ContainerStarted","Data":"742616ebcd4d4b72026b21a5452a72512fbeb192eda871678a072c193b51c2ae"}
Oct 30 01:10:21.956265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.955725    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:21.983257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:21.978810    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" podStartSLOduration=1.978790308 podStartE2EDuration="1.978790308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:21.954563208 +0000 UTC m=+1902.025887066" watchObservedRunningTime="2026-10-30 01:10:21.978790308 +0000 UTC m=+1902.050114145"
Oct 30 01:10:22.629030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:22.628995    4576 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:10:22.629319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:22.629292    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:10:22.787258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:22.781378    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" event={"ID":"9d55436d-2557-4b59-8409-6739cd1b4551","Type":"ContainerStarted","Data":"9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1"}
Oct 30 01:10:22.930413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:22.930324    4576 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:10:22.931171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:22.930995    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 01:10:23.145522 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.145488    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:23.223545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.223226    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-openshift-ca\") pod \"0b08a05f-6134-4a42-a611-2a14c7ec0628\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") "
Oct 30 01:10:23.223545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.223541    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b08a05f-6134-4a42-a611-2a14c7ec0628-kube-api-access-z77xv\" (UniqueName: \"kubernetes.io/projected/0b08a05f-6134-4a42-a611-2a14c7ec0628-kube-api-access-z77xv\") pod \"0b08a05f-6134-4a42-a611-2a14c7ec0628\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") "
Oct 30 01:10:23.223808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.223637    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-xks-ca\") pod \"0b08a05f-6134-4a42-a611-2a14c7ec0628\" (UID: \"0b08a05f-6134-4a42-a611-2a14c7ec0628\") "
Oct 30 01:10:23.224168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.223985    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-openshift-ca" pod "0b08a05f-6134-4a42-a611-2a14c7ec0628" (UID: "0b08a05f-6134-4a42-a611-2a14c7ec0628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:23.224707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.224361    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:10:23.227938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.227374    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-xks-ca" pod "0b08a05f-6134-4a42-a611-2a14c7ec0628" (UID: "0b08a05f-6134-4a42-a611-2a14c7ec0628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:23.227938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.227880    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b08a05f-6134-4a42-a611-2a14c7ec0628-kube-api-access-z77xv" pod "0b08a05f-6134-4a42-a611-2a14c7ec0628" (UID: "0b08a05f-6134-4a42-a611-2a14c7ec0628"). InnerVolumeSpecName "kube-api-access-z77xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:23.329253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.328547    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b08a05f-6134-4a42-a611-2a14c7ec0628-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:10:23.329253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.328586    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z77xv\" (UniqueName: \"kubernetes.io/projected/0b08a05f-6134-4a42-a611-2a14c7ec0628-kube-api-access-z77xv\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:10:23.447059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.447018    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:23.628892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.628794    4576 patch_prober.go:29] interesting pod/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:10:23.628892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.628845    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:10:23.804260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.804124    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"
Oct 30 01:10:23.806616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.806441    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b" event={"ID":"0b08a05f-6134-4a42-a611-2a14c7ec0628","Type":"ContainerDied","Data":"0435c4e527e6ce88cf75fd99e9002347e7e29f7dc47003b1266d2a1a8309b140"}
Oct 30 01:10:23.806616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.806496    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0435c4e527e6ce88cf75fd99e9002347e7e29f7dc47003b1266d2a1a8309b140"
Oct 30 01:10:23.867055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.867026    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:23.919803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.919715    4576 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:10:23.919803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:23.919774    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 01:10:24.633815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:24.633783    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:24.634106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:24.634081    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:24.634844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:24.634819    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:10:24.908616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:24.908509    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:24.920035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:24.920008    4576 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:10:24.920160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:24.920072    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 01:10:25.750813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:25.750778    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:25.862314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:25.862277    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:25.919721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:25.919679    4576 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:10:25.920105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:25.919748    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 01:10:26.513997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:26.513960    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:26.920062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:26.919981    4576 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:10:26.920062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:26.920043    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 01:10:27.920082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:27.920040    4576 patch_prober.go:29] interesting pod/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 01:10:27.920597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:27.920102    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 01:10:28.048801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:28.048762    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:28.164762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:28.164720    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:28.274480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:28.274435    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:28.928314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:28.928277    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:28.928858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:28.928834    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:28.930272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:28.930221    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:10:29.990536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:29.990507    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:31.087187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:31.087154    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:31.199071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:31.199037    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:31.671259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:31.671176    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:33.365245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:33.365141    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:34.947290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:34.947252    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:35.199115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.199018    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:35.849832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.849798    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"]
Oct 30 01:10:35.850490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.850476    4576 cpu_manager.go:414] "containerMap: removing container" podUID="0b08a05f-6134-4a42-a611-2a14c7ec0628" containerName="revoke-keys"
Oct 30 01:10:35.850530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.850492    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b08a05f-6134-4a42-a611-2a14c7ec0628" containerName="revoke-keys"
Oct 30 01:10:35.850649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.850639    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b08a05f-6134-4a42-a611-2a14c7ec0628" containerName="revoke-keys"
Oct 30 01:10:35.854705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.854678    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:35.881969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.881944    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"]
Oct 30 01:10:35.890710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.890686    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-d0ae0d\"/\"default-dockercfg-cv4f2\""
Oct 30 01:10:35.890857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.890806    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-d0ae0d\"/\"auth-test-model-d0ae0d-kserve-self-signed-certs\""
Oct 30 01:10:35.890857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.890806    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-d0ae0d\"/\"openshift-service-ca.crt\""
Oct 30 01:10:35.890934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.890889    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-d0ae0d\"/\"kube-root-ca.crt\""
Oct 30 01:10:35.925107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.925072    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:35.986948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.986904    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-dshm\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:35.986948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.986942    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-home\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:35.987559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.986970    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-tmp-dir\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:35.987559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.987174    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc98774-772b-4f71-99df-48d0b53a35b4-tls-certs\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:35.987559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.987301    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfrr\" (UniqueName: \"kubernetes.io/projected/9dc98774-772b-4f71-99df-48d0b53a35b4-kube-api-access-dtfrr\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:35.987559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:35.987371    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-model-cache\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.095195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.095152    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-home\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.097071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.097028    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-tmp-dir\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.097581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.097554    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc98774-772b-4f71-99df-48d0b53a35b4-tls-certs\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.097813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.097794    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfrr\" (UniqueName: \"kubernetes.io/projected/9dc98774-772b-4f71-99df-48d0b53a35b4-kube-api-access-dtfrr\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.098009 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.097981    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-model-cache\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.100449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.098399    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-dshm\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.111744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.095843    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-home\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.111744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.101255    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-model-cache\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.111744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.101495    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-tmp-dir\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.117612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.117582    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-dshm\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.133035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.133005    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc98774-772b-4f71-99df-48d0b53a35b4-tls-certs\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.147317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.139464    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfrr\" (UniqueName: \"kubernetes.io/projected/9dc98774-772b-4f71-99df-48d0b53a35b4-kube-api-access-dtfrr\") pod \"auth-test-model-d0ae0d-kserve-5847cc7557-z72lm\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.181262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.179644    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:36.512403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.512370    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"]
Oct 30 01:10:36.512953 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:10:36.512920    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dc98774_772b_4f71_99df_48d0b53a35b4.slice/crio-dca9e552153f2bba8d00f5d9a7ee0b8268f4a679cfc80bc7f3916fe69196b4f7 WatchSource:0}: Error finding container dca9e552153f2bba8d00f5d9a7ee0b8268f4a679cfc80bc7f3916fe69196b4f7: Status 404 returned error can't find the container with id dca9e552153f2bba8d00f5d9a7ee0b8268f4a679cfc80bc7f3916fe69196b4f7
Oct 30 01:10:36.658776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.658739    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:36.797451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.797341    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:36.902034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.901994    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm" event={"ID":"9dc98774-772b-4f71-99df-48d0b53a35b4","Type":"ContainerStarted","Data":"232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420"}
Oct 30 01:10:36.902187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.902044    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm" event={"ID":"9dc98774-772b-4f71-99df-48d0b53a35b4","Type":"ContainerStarted","Data":"dca9e552153f2bba8d00f5d9a7ee0b8268f4a679cfc80bc7f3916fe69196b4f7"}
Oct 30 01:10:36.946460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:36.946402    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm" podStartSLOduration=1.946387161 podStartE2EDuration="1.946387161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:36.944482609 +0000 UTC m=+1917.015806445" watchObservedRunningTime="2026-10-30 01:10:36.946387161 +0000 UTC m=+1917.017710996"
Oct 30 01:10:37.181454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:37.181354    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:38.324893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:38.324859    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:39.262296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:39.262222    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:39.991632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:39.991590    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:41.492527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:41.492492    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:42.811374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.811339    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"]
Oct 30 01:10:42.816508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.816484    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.848067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.848022    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bc6092-openshift-default-dockercfg-zrj92\""
Oct 30 01:10:42.852922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.852888    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"]
Oct 30 01:10:42.943762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.943726    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pscl\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-kube-api-access-7pscl\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.943948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.943778    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-certs\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.943948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.943805    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.944076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.943958    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-token\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.944135 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.944098    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-envoy\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.944197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.944140    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-podinfo\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.944300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.944270    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-data\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.944380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.944362    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istiod-ca-cert\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:42.944437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:42.944423    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-credential-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045366    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045451    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-token\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045553    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-envoy\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045582    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-podinfo\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045641    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-data\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045757    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istiod-ca-cert\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045836    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-credential-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045853    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045910    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pscl\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-kube-api-access-7pscl\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.045992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.045960    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-certs\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.046135 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.046111    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-credential-socket\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.046387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.046357    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-certs\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.046472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.046407    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-data\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.047064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.047041    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istiod-ca-cert\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.048388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.048364    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-envoy\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.048696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.048675    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-podinfo\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.067088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.067022    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-token\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.067424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.067404    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pscl\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-kube-api-access-7pscl\") pod \"e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") " pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.078268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.078208    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:43.130812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.130780    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:43.475375 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:10:43.475342    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc735cf_9b64_4fcc_9f94_19fc59baf9c1.slice/crio-517c795daa1a0a345ed68399e4636ffed79cc9c5886ee7cf16fe676e864b0643 WatchSource:0}: Error finding container 517c795daa1a0a345ed68399e4636ffed79cc9c5886ee7cf16fe676e864b0643: Status 404 returned error can't find the container with id 517c795daa1a0a345ed68399e4636ffed79cc9c5886ee7cf16fe676e864b0643
Oct 30 01:10:43.478311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.478283    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"]
Oct 30 01:10:43.479457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.479422    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:43.479552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.479500    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:43.479552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.479545    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:43.538750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.538720    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"]
Oct 30 01:10:43.544824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.544796    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.564961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.564913    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-42d56b-openshift-default-dockercfg-7msn4\""
Oct 30 01:10:43.586261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.571884    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"]
Oct 30 01:10:43.655957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.655909    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.656176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.655996    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-credential-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.656176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.656040    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-data\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.656319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.656226    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgdpl\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-kube-api-access-fgdpl\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.656374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.656321    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-certs\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.656428 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.656392    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istiod-ca-cert\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.656485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.656453    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-podinfo\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.656536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.656520    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-envoy\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.656594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.656560    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-token\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758363    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-credential-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758410    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-data\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758509    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgdpl\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-kube-api-access-fgdpl\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758561    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-certs\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758597    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istiod-ca-cert\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758657    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-podinfo\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758831    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-credential-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758852    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-data\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758853    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-envoy\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.758917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758891    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-certs\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.759122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.758939    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-token\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.759122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.759085    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.759533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.759483    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-socket\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.759533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.759521    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istiod-ca-cert\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.761314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.761294    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-envoy\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.762048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.762027    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-podinfo\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.779171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.779141    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-token\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.779434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.779415    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgdpl\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-kube-api-access-fgdpl\") pod \"e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") " pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.892337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.892302    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:43.974452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.970445    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" event={"ID":"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1","Type":"ContainerStarted","Data":"30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d"}
Oct 30 01:10:43.974452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:43.970512    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" event={"ID":"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1","Type":"ContainerStarted","Data":"517c795daa1a0a345ed68399e4636ffed79cc9c5886ee7cf16fe676e864b0643"}
Oct 30 01:10:44.033950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.033824    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podStartSLOduration=2.033804767 podStartE2EDuration="2.033804767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:44.030012987 +0000 UTC m=+1924.101336824" watchObservedRunningTime="2026-10-30 01:10:44.033804767 +0000 UTC m=+1924.105128607"
Oct 30 01:10:44.131536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.131505    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:44.133197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.133169    4576 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:10:44.133306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.133245    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 01:10:44.229984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.229795    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"]
Oct 30 01:10:44.231761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.231727    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:44.231891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.231801    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:44.231891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.231833    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:10:44.649215 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:44.649132    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:45.001207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:45.001161    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" event={"ID":"8aaa2da1-a942-426f-8fb5-79d3d5be7b84","Type":"ContainerStarted","Data":"be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b"}
Oct 30 01:10:45.001702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:45.001221    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" event={"ID":"8aaa2da1-a942-426f-8fb5-79d3d5be7b84","Type":"ContainerStarted","Data":"5a365acbfcd3bb72d7b51ea036d9e9d1d229d469339f67f66c03eeb0dd4201f5"}
Oct 30 01:10:45.101595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:45.101526    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" podStartSLOduration=2.101499556 podStartE2EDuration="2.101499556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:45.058808238 +0000 UTC m=+1925.130132072" watchObservedRunningTime="2026-10-30 01:10:45.101499556 +0000 UTC m=+1925.172823391"
Oct 30 01:10:45.134662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:45.134624    4576 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:10:45.134824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:45.134687    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 01:10:45.909259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:45.898690    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:45.909259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:45.899978    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 01:10:45.909259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:45.900033    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 01:10:46.133109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:46.133051    4576 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:10:46.133646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:46.133110    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 01:10:46.206673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:46.188788    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:46.220274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:46.216938    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:46.236674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:46.236647    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:46.894692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:46.894652    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 01:10:46.895092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:46.895007    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 01:10:47.049405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:47.049368    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:10:47.141326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:47.131612    4576 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:10:47.141326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:47.131677    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 01:10:47.849186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:47.849150    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:47.894677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:47.894634    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 01:10:47.894858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:47.894701    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 01:10:48.133694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:48.133611    4576 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:10:48.134062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:48.134035    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 01:10:48.436019 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:10:48.435856    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:10:48.893463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:48.893428    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 01:10:48.893749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:48.893720    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 01:10:49.132056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:49.132010    4576 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:10:49.132401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:49.132357    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 01:10:49.503146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:49.503107    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:49.615260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:49.613004    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:49.898746 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:49.898655    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 01:10:49.898746 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:49.898717    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 01:10:50.032497 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:50.032457    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"]
Oct 30 01:10:50.044247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:50.044198    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc6e6801-q5v7k"]
Oct 30 01:10:50.113096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:50.113061    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b286824-c144-4779-ba31-7337eb04df94" path="/var/lib/kubelet/pods/3b286824-c144-4779-ba31-7337eb04df94/volumes"
Oct 30 01:10:50.132093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:50.132052    4576 patch_prober.go:29] interesting pod/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 01:10:50.132285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:50.132117    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 01:10:50.892997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:50.892961    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 01:10:50.893416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:50.893020    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 01:10:51.138358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:51.138199    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:51.139723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:51.139700    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:51.141181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:51.141161    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:10:51.174140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:51.174069    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:51.913258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:51.912981    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:51.917255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:51.915018    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:51.935636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:51.935602    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:10:53.317771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:53.317728    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:54.816706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:54.816660    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:56.294870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:56.294835    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:57.973260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:57.968643    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:10:59.551084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:10:59.551049    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:01.065407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:01.065357    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:02.506365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:02.506322    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:02.957115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:02.957077    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"]
Oct 30 01:11:02.958039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:02.958003    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy" containerID="cri-o://879d87154e408a4e288b31d9ff675a57001d4b9b31a097884a6a695418ebd1dd" gracePeriod=30
Oct 30 01:11:03.296004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:03.295919    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:03.425760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:03.425731    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"]
Oct 30 01:11:03.429593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:03.429565    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ca20f-g7g9b"]
Oct 30 01:11:03.432727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:03.432695    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:03.543293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:03.543190    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:03.906144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:03.906054    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:04.110709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:04.110666    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b08a05f-6134-4a42-a611-2a14c7ec0628" path="/var/lib/kubelet/pods/0b08a05f-6134-4a42-a611-2a14c7ec0628/volumes"
Oct 30 01:11:04.186640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:04.186562    4576 generic.go:356] "Generic (PLEG): container finished" podID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerID="e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c" exitCode=0
Oct 30 01:11:04.186797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:04.186633    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" event={"ID":"4028e239-916a-4474-9da9-bb2f47dfb84d","Type":"ContainerDied","Data":"e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c"}
Oct 30 01:11:04.187138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:04.187124    4576 scope.go:122] "RemoveContainer" containerID="e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c"
Oct 30 01:11:04.189049 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:04.189024    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:11:04.189117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:04.189093    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:11:04.189156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:04.189125    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:11:04.223512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:04.223482    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:05.202949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:05.202917    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" event={"ID":"4028e239-916a-4474-9da9-bb2f47dfb84d","Type":"ContainerStarted","Data":"44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c"}
Oct 30 01:11:05.251164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:05.251121    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:06.156404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:06.156364    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:11:06.156586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:06.156529    4576 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:11:06.156636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:06.156578    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 01:11:06.794275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:06.789779    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:07.155837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:07.155740    4576 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:11:07.155837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:07.155804    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 01:11:08.156042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.155998    4576 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:11:08.156453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.156059    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 01:11:08.230188 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.230155    4576 generic.go:356] "Generic (PLEG): container finished" podID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerID="879d87154e408a4e288b31d9ff675a57001d4b9b31a097884a6a695418ebd1dd" exitCode=0
Oct 30 01:11:08.230430 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.230415    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" event={"ID":"100b43f2-ffa5-4095-ab1a-bbcb77417026","Type":"ContainerDied","Data":"879d87154e408a4e288b31d9ff675a57001d4b9b31a097884a6a695418ebd1dd"}
Oct 30 01:11:08.316975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.316947    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 01:11:08.356510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.356474    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:08.429330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429052    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-token\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-token\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.429330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429106    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100b43f2-ffa5-4095-ab1a-bbcb77417026-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100b43f2-ffa5-4095-ab1a-bbcb77417026-istiod-ca-cert\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.429330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429163    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-data\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.429330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429296    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-certs\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.430905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429352    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-credential-socket\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.430905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429472    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-envoy\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.430905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429586    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-podinfo\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.430905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429659    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-socket\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.430905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.429805    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-kube-api-access-p44mp\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-kube-api-access-p44mp\") pod \"100b43f2-ffa5-4095-ab1a-bbcb77417026\" (UID: \"100b43f2-ffa5-4095-ab1a-bbcb77417026\") "
Oct 30 01:11:08.444987 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.443946    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-kube-api-access-p44mp" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "kube-api-access-p44mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:08.447891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.447814    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-socket" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:08.448055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.447934    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-data" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:08.448151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.448128    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-certs" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:08.448443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.448419    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100b43f2-ffa5-4095-ab1a-bbcb77417026-istiod-ca-cert" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:08.448690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.448669    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-credential-socket" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:08.449090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.449070    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-token" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:08.451707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.451665    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-podinfo" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:11:08.455064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.455035    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-envoy" pod "100b43f2-ffa5-4095-ab1a-bbcb77417026" (UID: "100b43f2-ffa5-4095-ab1a-bbcb77417026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:08.532388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532282    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44mp\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-kube-api-access-p44mp\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.532388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532320    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.532388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532334    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100b43f2-ffa5-4095-ab1a-bbcb77417026-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.532388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532346    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.532388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532359    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.532388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532373    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.532388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532386    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.532779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532398    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100b43f2-ffa5-4095-ab1a-bbcb77417026-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.532779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:08.532411    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100b43f2-ffa5-4095-ab1a-bbcb77417026-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:09.155845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:09.155811    4576 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:11:09.156021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:09.155891    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 01:11:09.239157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:09.239129    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"
Oct 30 01:11:09.239663 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:09.239168    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt" event={"ID":"100b43f2-ffa5-4095-ab1a-bbcb77417026","Type":"ContainerDied","Data":"b10516fb7bd0fd471ac7de0439ca1d2624205cf06709e037f462b043994b3f40"}
Oct 30 01:11:09.239663 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:09.239274    4576 scope.go:122] "RemoveContainer" containerID="879d87154e408a4e288b31d9ff675a57001d4b9b31a097884a6a695418ebd1dd"
Oct 30 01:11:09.325546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:09.325512    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"]
Oct 30 01:11:09.339738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:09.339701    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8cf33a-openshift-default-79df9cd7c5-qrdpt"]
Oct 30 01:11:09.893101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:09.892941    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:10.054043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.054005    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:10.118830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.118796    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" path="/var/lib/kubelet/pods/100b43f2-ffa5-4095-ab1a-bbcb77417026/volumes"
Oct 30 01:11:10.155778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.155698    4576 patch_prober.go:29] interesting pod/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 01:11:10.155778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.155756    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 01:11:10.176677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.176639    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:10.338478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.338445    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"]
Oct 30 01:11:10.339050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.339016    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy" containerID="cri-o://eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870" gracePeriod=30
Oct 30 01:11:10.403600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.403315    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:10.910062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.910023    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"]
Oct 30 01:11:10.910817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.910791    4576 cpu_manager.go:414] "containerMap: removing container" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy"
Oct 30 01:11:10.910817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.910811    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy"
Oct 30 01:11:10.911043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.910979    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="100b43f2-ffa5-4095-ab1a-bbcb77417026" containerName="istio-proxy"
Oct 30 01:11:10.916066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.916040    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:10.945987 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.945953    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"]
Oct 30 01:11:10.972616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.972580    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:10.972767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.972648    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:11:10.992090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.992061    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:10.992368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.992217    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq82\" (UniqueName: \"kubernetes.io/projected/7005b725-d30f-4079-9eda-3eb35e70f796-kube-api-access-vvq82\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:10.992457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:10.992369    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:11.093965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.093928    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:11.094142 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.094047    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq82\" (UniqueName: \"kubernetes.io/projected/7005b725-d30f-4079-9eda-3eb35e70f796-kube-api-access-vvq82\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:11.094142 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.094117    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:11.094825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.094805    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:11.096470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.096452    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:11.115803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.115773    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq82\" (UniqueName: \"kubernetes.io/projected/7005b725-d30f-4079-9eda-3eb35e70f796-kube-api-access-vvq82\") pod \"maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:11.161356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.161227    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:11:11.161799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.161780    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:11:11.162516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.162496    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:11:11.230715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.230679    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:11.558837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.558805    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"]
Oct 30 01:11:11.560283 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:11:11.560251    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7005b725_d30f_4079_9eda_3eb35e70f796.slice/crio-4fd0920882e6bfa2db5121293ae501cb60dffd8af0e3d461234eb59215efa9b0 WatchSource:0}: Error finding container 4fd0920882e6bfa2db5121293ae501cb60dffd8af0e3d461234eb59215efa9b0: Status 404 returned error can't find the container with id 4fd0920882e6bfa2db5121293ae501cb60dffd8af0e3d461234eb59215efa9b0
Oct 30 01:11:11.730527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:11.730494    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:12.266892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:12.266856    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28" event={"ID":"7005b725-d30f-4079-9eda-3eb35e70f796","Type":"ContainerStarted","Data":"6418d1eacdcd8bc63a2b40e9a1a165c0abe7e6e35ac0449b494acd0be66ed00e"}
Oct 30 01:11:12.267091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:12.266918    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28" event={"ID":"7005b725-d30f-4079-9eda-3eb35e70f796","Type":"ContainerStarted","Data":"4fd0920882e6bfa2db5121293ae501cb60dffd8af0e3d461234eb59215efa9b0"}
Oct 30 01:11:12.311809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:12.311751    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28" podStartSLOduration=2.311736593 podStartE2EDuration="2.311736593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:12.309302193 +0000 UTC m=+1952.380626029" watchObservedRunningTime="2026-10-30 01:11:12.311736593 +0000 UTC m=+1952.383060427"
Oct 30 01:11:12.963611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:12.963577    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"]
Oct 30 01:11:12.968044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:12.968020    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.002868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:12.998441    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0680fa-openshift-default-dockercfg-gxjnz\""
Oct 30 01:11:13.002868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.002813    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"]
Oct 30 01:11:13.029343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029169    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-data\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.029343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029288    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-certs\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.029601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029415    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.029601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029496    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-credential-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.029601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029535    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-token\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.029601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029564    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5pq\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-kube-api-access-9x5pq\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.029840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029706    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-envoy\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.029840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029752    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-podinfo\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.029840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.029779    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6000abc5-dcfc-4c59-a169-037cfb59be1b-istiod-ca-cert\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.131396 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.131359    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-data\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.131604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.131430    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-certs\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.131604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.131544    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.131844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.131820    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-data\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.131844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.131838    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.131965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.131825    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-credential-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.131965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.131896    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-token\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.131965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.131927    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5pq\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-kube-api-access-9x5pq\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.132113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.132053    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-credential-socket\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.132113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.132092    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-envoy\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.132218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.132158    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-podinfo\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.132312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.132288    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6000abc5-dcfc-4c59-a169-037cfb59be1b-istiod-ca-cert\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.132637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.132606    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-certs\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.133047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.133022    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6000abc5-dcfc-4c59-a169-037cfb59be1b-istiod-ca-cert\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.134583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.134559    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-envoy\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.135076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.135056    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-podinfo\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.153097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.153067    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-token\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.153337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.153311    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5pq\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-kube-api-access-9x5pq\") pod \"e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") " pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.278418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.278328    4576 generic.go:356] "Generic (PLEG): container finished" podID="7005b725-d30f-4079-9eda-3eb35e70f796" containerID="6418d1eacdcd8bc63a2b40e9a1a165c0abe7e6e35ac0449b494acd0be66ed00e" exitCode=0
Oct 30 01:11:13.278594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.278504    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28" event={"ID":"7005b725-d30f-4079-9eda-3eb35e70f796","Type":"ContainerDied","Data":"6418d1eacdcd8bc63a2b40e9a1a165c0abe7e6e35ac0449b494acd0be66ed00e"}
Oct 30 01:11:13.282647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.282624    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:13.442732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.442686    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:13.622884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.622853    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"]
Oct 30 01:11:13.623407 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:11:13.623381    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6000abc5_dcfc_4c59_a169_037cfb59be1b.slice/crio-e2e53286c1891cc6a2179af69330dc82034bc215e180fce3f2703e1568eefbec WatchSource:0}: Error finding container e2e53286c1891cc6a2179af69330dc82034bc215e180fce3f2703e1568eefbec: Status 404 returned error can't find the container with id e2e53286c1891cc6a2179af69330dc82034bc215e180fce3f2703e1568eefbec
Oct 30 01:11:13.626286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.626250    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:11:13.626387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.626337    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:11:13.626451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:13.626386    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:11:14.303051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.303011    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" event={"ID":"6000abc5-dcfc-4c59-a169-037cfb59be1b","Type":"ContainerStarted","Data":"4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b"}
Oct 30 01:11:14.303514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.303079    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" event={"ID":"6000abc5-dcfc-4c59-a169-037cfb59be1b","Type":"ContainerStarted","Data":"e2e53286c1891cc6a2179af69330dc82034bc215e180fce3f2703e1568eefbec"}
Oct 30 01:11:14.348218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.348151    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" podStartSLOduration=2.348135102 podStartE2EDuration="2.348135102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:14.346092787 +0000 UTC m=+1954.417416622" watchObservedRunningTime="2026-10-30 01:11:14.348135102 +0000 UTC m=+1954.419458937"
Oct 30 01:11:14.508902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.508876    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:14.562348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.562271    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7005b725-d30f-4079-9eda-3eb35e70f796-kube-api-access-vvq82\" (UniqueName: \"kubernetes.io/projected/7005b725-d30f-4079-9eda-3eb35e70f796-kube-api-access-vvq82\") pod \"7005b725-d30f-4079-9eda-3eb35e70f796\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") "
Oct 30 01:11:14.562640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.562611    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-openshift-ca\") pod \"7005b725-d30f-4079-9eda-3eb35e70f796\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") "
Oct 30 01:11:14.562748 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.562699    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-xks-ca\") pod \"7005b725-d30f-4079-9eda-3eb35e70f796\" (UID: \"7005b725-d30f-4079-9eda-3eb35e70f796\") "
Oct 30 01:11:14.563343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.563024    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-openshift-ca" pod "7005b725-d30f-4079-9eda-3eb35e70f796" (UID: "7005b725-d30f-4079-9eda-3eb35e70f796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:14.563464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.563418    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:14.580296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.568752    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-xks-ca" pod "7005b725-d30f-4079-9eda-3eb35e70f796" (UID: "7005b725-d30f-4079-9eda-3eb35e70f796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:14.580296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.568865    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7005b725-d30f-4079-9eda-3eb35e70f796-kube-api-access-vvq82" pod "7005b725-d30f-4079-9eda-3eb35e70f796" (UID: "7005b725-d30f-4079-9eda-3eb35e70f796"). InnerVolumeSpecName "kube-api-access-vvq82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:14.664985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.664952    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7005b725-d30f-4079-9eda-3eb35e70f796-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:14.664985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:14.664982    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvq82\" (UniqueName: \"kubernetes.io/projected/7005b725-d30f-4079-9eda-3eb35e70f796-kube-api-access-vvq82\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.025654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.025618    4576 patch_prober.go:29] interesting pod/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:11:15.025829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.025678    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:11:15.192293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.192262    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:15.288410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.288312    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:15.291385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.291356    4576 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 01:11:15.293042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.291598    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 01:11:15.312710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.312686    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"
Oct 30 01:11:15.313305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.312704    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28" event={"ID":"7005b725-d30f-4079-9eda-3eb35e70f796","Type":"ContainerDied","Data":"4fd0920882e6bfa2db5121293ae501cb60dffd8af0e3d461234eb59215efa9b0"}
Oct 30 01:11:15.313305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.312753    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fd0920882e6bfa2db5121293ae501cb60dffd8af0e3d461234eb59215efa9b0"
Oct 30 01:11:15.565946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.565325    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:15.720198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.719860    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.804942    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36025db8-f787-4abf-be74-d6aab77bc769-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36025db8-f787-4abf-be74-d6aab77bc769-istio-podinfo\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.805219    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-credential-socket\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.805499    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-kube-api-access-4vh7s\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-kube-api-access-4vh7s\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.805723    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-envoy\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.805815    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-certs\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.806084    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-data\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.809624    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-credential-socket" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.809915    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-certs" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.810769    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-kube-api-access-4vh7s" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "kube-api-access-4vh7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.811423    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-istio-token\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-istio-token\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.811508    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-socket\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.812055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.811571    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36025db8-f787-4abf-be74-d6aab77bc769-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36025db8-f787-4abf-be74-d6aab77bc769-istiod-ca-cert\") pod \"36025db8-f787-4abf-be74-d6aab77bc769\" (UID: \"36025db8-f787-4abf-be74-d6aab77bc769\") "
Oct 30 01:11:15.813116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.812825    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.813116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.812858    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vh7s\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-kube-api-access-4vh7s\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.813116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.812873    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.813385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.813307    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-socket" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:15.813451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.813374    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-data" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:15.813999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.813690    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36025db8-f787-4abf-be74-d6aab77bc769-istiod-ca-cert" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:15.822389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.822295    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36025db8-f787-4abf-be74-d6aab77bc769-istio-podinfo" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:11:15.823168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.822431    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-envoy" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:15.824142 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.824105    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-istio-token" pod "36025db8-f787-4abf-be74-d6aab77bc769" (UID: "36025db8-f787-4abf-be74-d6aab77bc769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:15.914277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.914115    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.914277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.914143    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36025db8-f787-4abf-be74-d6aab77bc769-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.914277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.914153    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.914277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.914161    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36025db8-f787-4abf-be74-d6aab77bc769-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.914277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.914169    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36025db8-f787-4abf-be74-d6aab77bc769-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:15.914277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:15.914176    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36025db8-f787-4abf-be74-d6aab77bc769-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:16.006423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.006377    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:16.293418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.287665    4576 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 01:11:16.293418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.287726    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 01:11:16.325141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.324567    4576 generic.go:356] "Generic (PLEG): container finished" podID="36025db8-f787-4abf-be74-d6aab77bc769" containerID="eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870" exitCode=0
Oct 30 01:11:16.325141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.324701    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"
Oct 30 01:11:16.325141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.324720    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" event={"ID":"36025db8-f787-4abf-be74-d6aab77bc769","Type":"ContainerDied","Data":"eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870"}
Oct 30 01:11:16.325141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.324780    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk" event={"ID":"36025db8-f787-4abf-be74-d6aab77bc769","Type":"ContainerDied","Data":"8a4473a70dba938e0e63475bf2f8762f833b2d365a76dd08f8a04b809aadaf2e"}
Oct 30 01:11:16.325141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.324814    4576 scope.go:122] "RemoveContainer" containerID="eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870"
Oct 30 01:11:16.340030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.339985    4576 scope.go:122] "RemoveContainer" containerID="eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870"
Oct 30 01:11:16.340455 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:11:16.340423    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870\": container with ID starting with eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870 not found: ID does not exist" containerID="eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870"
Oct 30 01:11:16.340545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.340468    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870"} err="failed to get container status \"eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870\": rpc error: code = NotFound desc = could not find container \"eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870\": container with ID starting with eab6ce3d9a95d8ec57c1445bc77573cf163dd0cc0016c8e6287efdf689b26870 not found: ID does not exist"
Oct 30 01:11:16.406757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.406679    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"]
Oct 30 01:11:16.414832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.414797    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2ca20f-openshift-default-5f667c45f9-hqrdk"]
Oct 30 01:11:16.528609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:16.527612    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:17.155358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.155329    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:17.262685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.262645    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:17.286261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.284644    4576 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 01:11:17.286261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.284703    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 01:11:17.506532 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.506495    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"]
Oct 30 01:11:17.507524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.507503    4576 cpu_manager.go:414] "containerMap: removing container" podUID="7005b725-d30f-4079-9eda-3eb35e70f796" containerName="revoke-keys"
Oct 30 01:11:17.507524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.507525    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7005b725-d30f-4079-9eda-3eb35e70f796" containerName="revoke-keys"
Oct 30 01:11:17.507688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.507550    4576 cpu_manager.go:414] "containerMap: removing container" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy"
Oct 30 01:11:17.507688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.507556    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy"
Oct 30 01:11:17.507688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.507659    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="36025db8-f787-4abf-be74-d6aab77bc769" containerName="istio-proxy"
Oct 30 01:11:17.507688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.507676    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="7005b725-d30f-4079-9eda-3eb35e70f796" containerName="revoke-keys"
Oct 30 01:11:17.513011 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.512986    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.530854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.530818    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"]
Oct 30 01:11:17.538579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.538543    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:17.538834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.538801    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:11:17.553932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.553708    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1416378-ba86-4a58-b135-04644376b588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.554105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.554050    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1416378-ba86-4a58-b135-04644376b588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.554209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.554169    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqbg\" (UniqueName: \"kubernetes.io/projected/f1416378-ba86-4a58-b135-04644376b588-kube-api-access-4dqbg\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.656029 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.655987    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqbg\" (UniqueName: \"kubernetes.io/projected/f1416378-ba86-4a58-b135-04644376b588-kube-api-access-4dqbg\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.656209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.656086    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1416378-ba86-4a58-b135-04644376b588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.656333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.656314    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1416378-ba86-4a58-b135-04644376b588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.656978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.656953    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1416378-ba86-4a58-b135-04644376b588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.658984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.658958    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1416378-ba86-4a58-b135-04644376b588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.699260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.699148    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqbg\" (UniqueName: \"kubernetes.io/projected/f1416378-ba86-4a58-b135-04644376b588-kube-api-access-4dqbg\") pod \"maas-api-revoke-sub-c5d2ba8d-cnkt4\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:17.835705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:17.835608    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:18.203441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.203399    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36025db8-f787-4abf-be74-d6aab77bc769" path="/var/lib/kubelet/pods/36025db8-f787-4abf-be74-d6aab77bc769/volumes"
Oct 30 01:11:18.206560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.206444    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"]
Oct 30 01:11:18.212213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.212192    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"]
Oct 30 01:11:18.213028 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:11:18.213005    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1416378_ba86_4a58_b135_04644376b588.slice/crio-b37a3a484e846befa8d156cb528352f5bd0fba4d5138517ca819f2259c007cc8 WatchSource:0}: Error finding container b37a3a484e846befa8d156cb528352f5bd0fba4d5138517ca819f2259c007cc8: Status 404 returned error can't find the container with id b37a3a484e846befa8d156cb528352f5bd0fba4d5138517ca819f2259c007cc8
Oct 30 01:11:18.218110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.218080    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.226279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.226225    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"]
Oct 30 01:11:18.288434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.288394    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7zqn\" (UniqueName: \"kubernetes.io/projected/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-kube-api-access-q7zqn\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.289701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.289673    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.290547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.290527    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.291271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.289430    4576 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 01:11:18.291427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.291403    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 01:11:18.346005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.345970    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4" event={"ID":"f1416378-ba86-4a58-b135-04644376b588","Type":"ContainerStarted","Data":"b37a3a484e846befa8d156cb528352f5bd0fba4d5138517ca819f2259c007cc8"}
Oct 30 01:11:18.393260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.392728    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.393260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.392859    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.393260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.393002    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7zqn\" (UniqueName: \"kubernetes.io/projected/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-kube-api-access-q7zqn\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.400593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.400004    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.402727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.402337    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.410888 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:11:18.410845    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:11:18.419922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.419895    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7zqn\" (UniqueName: \"kubernetes.io/projected/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-kube-api-access-q7zqn\") pod \"maas-api-revoke-sub-4c30fdad-qrcmv\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.548252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.548131    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:18.900449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.898866    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"]
Oct 30 01:11:18.901147 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:11:18.901103    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b9c3dc_6f68_4ebc_9632_b920bcd50b09.slice/crio-0bf274afb75401c98d958fd5f4b4154d4d44202010de5346336df86bf817a76c WatchSource:0}: Error finding container 0bf274afb75401c98d958fd5f4b4154d4d44202010de5346336df86bf817a76c: Status 404 returned error can't find the container with id 0bf274afb75401c98d958fd5f4b4154d4d44202010de5346336df86bf817a76c
Oct 30 01:11:18.939599 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:18.939503    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:19.293720 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:19.293678    4576 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 01:11:19.294008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:19.293740    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 01:11:19.354741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:19.354709    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv" event={"ID":"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09","Type":"ContainerStarted","Data":"0bf274afb75401c98d958fd5f4b4154d4d44202010de5346336df86bf817a76c"}
Oct 30 01:11:19.356331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:19.356271    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4" event={"ID":"f1416378-ba86-4a58-b135-04644376b588","Type":"ContainerStarted","Data":"17ce80a3ad2b192867c4e409717bdea94a739951940288cc5edf42b35fcb3674"}
Oct 30 01:11:19.408564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:19.408517    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4" podStartSLOduration=2.4085030610000002 podStartE2EDuration="2.408503061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:19.406600125 +0000 UTC m=+1959.477923960" watchObservedRunningTime="2026-10-30 01:11:19.408503061 +0000 UTC m=+1959.479826909"
Oct 30 01:11:20.316368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.314758    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:20.316368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.315597    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:20.317290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.317199    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:11:20.333525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.333495    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:20.366816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.366780    4576 generic.go:356] "Generic (PLEG): container finished" podID="f9b9c3dc-6f68-4ebc-9632-b920bcd50b09" containerID="2b70252732fe4b61aeb0c83cfd9fca50fd9bac6ca637df20bdaf6a463544a839" exitCode=0
Oct 30 01:11:20.366991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.366892    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv" event={"ID":"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09","Type":"ContainerDied","Data":"2b70252732fe4b61aeb0c83cfd9fca50fd9bac6ca637df20bdaf6a463544a839"}
Oct 30 01:11:20.368591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.368566    4576 generic.go:356] "Generic (PLEG): container finished" podID="f1416378-ba86-4a58-b135-04644376b588" containerID="17ce80a3ad2b192867c4e409717bdea94a739951940288cc5edf42b35fcb3674" exitCode=0
Oct 30 01:11:20.368752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.368637    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4" event={"ID":"f1416378-ba86-4a58-b135-04644376b588","Type":"ContainerDied","Data":"17ce80a3ad2b192867c4e409717bdea94a739951940288cc5edf42b35fcb3674"}
Oct 30 01:11:20.463315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.458254    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:20.736879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:20.736824    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:21.517649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.517619    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:21.648683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.648648    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:21.659847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.659822    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:21.662292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.662209    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:21.762738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.762408    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1416378-ba86-4a58-b135-04644376b588-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1416378-ba86-4a58-b135-04644376b588-maas-api-xks-ca\") pod \"f1416378-ba86-4a58-b135-04644376b588\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") "
Oct 30 01:11:21.762738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.762521    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1416378-ba86-4a58-b135-04644376b588-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1416378-ba86-4a58-b135-04644376b588-maas-api-openshift-ca\") pod \"f1416378-ba86-4a58-b135-04644376b588\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") "
Oct 30 01:11:21.762738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.762609    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-kube-api-access-q7zqn\" (UniqueName: \"kubernetes.io/projected/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-kube-api-access-q7zqn\") pod \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") "
Oct 30 01:11:21.762738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.762667    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-openshift-ca\") pod \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") "
Oct 30 01:11:21.762738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.762699    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1416378-ba86-4a58-b135-04644376b588-kube-api-access-4dqbg\" (UniqueName: \"kubernetes.io/projected/f1416378-ba86-4a58-b135-04644376b588-kube-api-access-4dqbg\") pod \"f1416378-ba86-4a58-b135-04644376b588\" (UID: \"f1416378-ba86-4a58-b135-04644376b588\") "
Oct 30 01:11:21.763164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.762761    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-xks-ca\") pod \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\" (UID: \"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09\") "
Oct 30 01:11:21.765313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.765283    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-openshift-ca" pod "f9b9c3dc-6f68-4ebc-9632-b920bcd50b09" (UID: "f9b9c3dc-6f68-4ebc-9632-b920bcd50b09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:21.766025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.766003    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1416378-ba86-4a58-b135-04644376b588-maas-api-openshift-ca" pod "f1416378-ba86-4a58-b135-04644376b588" (UID: "f1416378-ba86-4a58-b135-04644376b588"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:21.766971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.766943    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-kube-api-access-q7zqn" pod "f9b9c3dc-6f68-4ebc-9632-b920bcd50b09" (UID: "f9b9c3dc-6f68-4ebc-9632-b920bcd50b09"). InnerVolumeSpecName "kube-api-access-q7zqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:21.767057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.766984    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1416378-ba86-4a58-b135-04644376b588-kube-api-access-4dqbg" pod "f1416378-ba86-4a58-b135-04644376b588" (UID: "f1416378-ba86-4a58-b135-04644376b588"). InnerVolumeSpecName "kube-api-access-4dqbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:21.767100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.767064    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1416378-ba86-4a58-b135-04644376b588-maas-api-xks-ca" pod "f1416378-ba86-4a58-b135-04644376b588" (UID: "f1416378-ba86-4a58-b135-04644376b588"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:21.768680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.768658    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-xks-ca" pod "f9b9c3dc-6f68-4ebc-9632-b920bcd50b09" (UID: "f9b9c3dc-6f68-4ebc-9632-b920bcd50b09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:21.865217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.865187    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1416378-ba86-4a58-b135-04644376b588-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:21.865217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.865219    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1416378-ba86-4a58-b135-04644376b588-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:21.865451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.865248    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7zqn\" (UniqueName: \"kubernetes.io/projected/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-kube-api-access-q7zqn\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:21.865451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.865258    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:21.865451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.865267    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqbg\" (UniqueName: \"kubernetes.io/projected/f1416378-ba86-4a58-b135-04644376b588-kube-api-access-4dqbg\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:21.865451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:21.865278    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:22.386362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.386331    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"
Oct 30 01:11:22.386559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.386361    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv" event={"ID":"f9b9c3dc-6f68-4ebc-9632-b920bcd50b09","Type":"ContainerDied","Data":"0bf274afb75401c98d958fd5f4b4154d4d44202010de5346336df86bf817a76c"}
Oct 30 01:11:22.386559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.386448    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf274afb75401c98d958fd5f4b4154d4d44202010de5346336df86bf817a76c"
Oct 30 01:11:22.387936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.387911    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4" event={"ID":"f1416378-ba86-4a58-b135-04644376b588","Type":"ContainerDied","Data":"b37a3a484e846befa8d156cb528352f5bd0fba4d5138517ca819f2259c007cc8"}
Oct 30 01:11:22.388071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.387963    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"
Oct 30 01:11:22.388071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.387954    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37a3a484e846befa8d156cb528352f5bd0fba4d5138517ca819f2259c007cc8"
Oct 30 01:11:22.672523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.672484    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:22.792650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.792614    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:22.915361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:22.915327    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:24.512943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:24.512910    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:25.997527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:25.997490    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:27.467644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:27.467609    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:28.397645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.397611    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"]
Oct 30 01:11:28.451066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.451034    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:28.970266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.970216    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"]
Oct 30 01:11:28.970946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.970929    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f9b9c3dc-6f68-4ebc-9632-b920bcd50b09" containerName="revoke-keys"
Oct 30 01:11:28.971007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.970948    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b9c3dc-6f68-4ebc-9632-b920bcd50b09" containerName="revoke-keys"
Oct 30 01:11:28.971007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.970970    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f1416378-ba86-4a58-b135-04644376b588" containerName="revoke-keys"
Oct 30 01:11:28.971007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.970975    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1416378-ba86-4a58-b135-04644376b588" containerName="revoke-keys"
Oct 30 01:11:28.971105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.971087    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b9c3dc-6f68-4ebc-9632-b920bcd50b09" containerName="revoke-keys"
Oct 30 01:11:28.971105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.971098    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1416378-ba86-4a58-b135-04644376b588" containerName="revoke-keys"
Oct 30 01:11:28.975453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.975434    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:28.994379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:28.994351    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"]
Oct 30 01:11:29.002078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.002039    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:29.002253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.002152    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:11:29.026372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.026335    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.026646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.026627    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l76f8\" (UniqueName: \"kubernetes.io/projected/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-kube-api-access-l76f8\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.026766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.026666    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.128036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.127999    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l76f8\" (UniqueName: \"kubernetes.io/projected/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-kube-api-access-l76f8\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.128270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.128051    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.128270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.128263    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.129049 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.129022    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.131019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.130996    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.148480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.148448    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l76f8\" (UniqueName: \"kubernetes.io/projected/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-kube-api-access-l76f8\") pod \"maas-api-revoke-sub-3a91efc1-7x56g\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.296098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.295975    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:29.333036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.333002    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:29.603092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:29.603058    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"]
Oct 30 01:11:29.603821 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:11:29.603794    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c3ae00_3ded_496c_86d4_b4bc3c9917a9.slice/crio-296b1d9dba19d3fdc6d681ce3aa7f3e2ce71d5e606d1dfa88b337dd0102ef4d8 WatchSource:0}: Error finding container 296b1d9dba19d3fdc6d681ce3aa7f3e2ce71d5e606d1dfa88b337dd0102ef4d8: Status 404 returned error can't find the container with id 296b1d9dba19d3fdc6d681ce3aa7f3e2ce71d5e606d1dfa88b337dd0102ef4d8
Oct 30 01:11:30.460699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:30.460659    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g" event={"ID":"69c3ae00-3ded-496c-86d4-b4bc3c9917a9","Type":"ContainerStarted","Data":"fde9ae916b7d4006be30bffadc9fdcc72307dfec29cca88a23ac981b73923cb4"}
Oct 30 01:11:30.461186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:30.460720    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g" event={"ID":"69c3ae00-3ded-496c-86d4-b4bc3c9917a9","Type":"ContainerStarted","Data":"296b1d9dba19d3fdc6d681ce3aa7f3e2ce71d5e606d1dfa88b337dd0102ef4d8"}
Oct 30 01:11:30.506769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:30.506714    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g" podStartSLOduration=2.50670073 podStartE2EDuration="2.50670073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:30.503959111 +0000 UTC m=+1970.575282945" watchObservedRunningTime="2026-10-30 01:11:30.50670073 +0000 UTC m=+1970.578024600"
Oct 30 01:11:30.791165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:30.791064    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:31.471088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:31.471032    4576 generic.go:356] "Generic (PLEG): container finished" podID="69c3ae00-3ded-496c-86d4-b4bc3c9917a9" containerID="fde9ae916b7d4006be30bffadc9fdcc72307dfec29cca88a23ac981b73923cb4" exitCode=0
Oct 30 01:11:31.471629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:31.471115    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g" event={"ID":"69c3ae00-3ded-496c-86d4-b4bc3c9917a9","Type":"ContainerDied","Data":"fde9ae916b7d4006be30bffadc9fdcc72307dfec29cca88a23ac981b73923cb4"}
Oct 30 01:11:32.264622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.264585    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:32.659288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.659250    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:32.779994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.779961    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-kube-api-access-l76f8\" (UniqueName: \"kubernetes.io/projected/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-kube-api-access-l76f8\") pod \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") "
Oct 30 01:11:32.780184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.780142    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-openshift-ca\") pod \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") "
Oct 30 01:11:32.780287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.780271    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-xks-ca\") pod \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\" (UID: \"69c3ae00-3ded-496c-86d4-b4bc3c9917a9\") "
Oct 30 01:11:32.780499 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.780475    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-openshift-ca" pod "69c3ae00-3ded-496c-86d4-b4bc3c9917a9" (UID: "69c3ae00-3ded-496c-86d4-b4bc3c9917a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:32.781142 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.781115    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:32.782668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.782640    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-kube-api-access-l76f8" pod "69c3ae00-3ded-496c-86d4-b4bc3c9917a9" (UID: "69c3ae00-3ded-496c-86d4-b4bc3c9917a9"). InnerVolumeSpecName "kube-api-access-l76f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:32.782761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.782708    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-xks-ca" pod "69c3ae00-3ded-496c-86d4-b4bc3c9917a9" (UID: "69c3ae00-3ded-496c-86d4-b4bc3c9917a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:32.882841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.882806    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:32.882841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:32.882835    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l76f8\" (UniqueName: \"kubernetes.io/projected/69c3ae00-3ded-496c-86d4-b4bc3c9917a9-kube-api-access-l76f8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:33.489904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:33.489868    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"
Oct 30 01:11:33.489904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:33.489875    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g" event={"ID":"69c3ae00-3ded-496c-86d4-b4bc3c9917a9","Type":"ContainerDied","Data":"296b1d9dba19d3fdc6d681ce3aa7f3e2ce71d5e606d1dfa88b337dd0102ef4d8"}
Oct 30 01:11:33.490110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:33.489927    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="296b1d9dba19d3fdc6d681ce3aa7f3e2ce71d5e606d1dfa88b337dd0102ef4d8"
Oct 30 01:11:33.976141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:33.976107    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:35.331533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:35.331498    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:35.554137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:35.554103    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:36.862098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:36.862046    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:37.028781 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:37.028745    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:38.723164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:38.723128    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:39.396883 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:39.396845    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:40.275787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.275749    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:40.665269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.665135    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:40.814220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.814175    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"]
Oct 30 01:11:40.815389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.815366    4576 cpu_manager.go:414] "containerMap: removing container" podUID="69c3ae00-3ded-496c-86d4-b4bc3c9917a9" containerName="revoke-keys"
Oct 30 01:11:40.815488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.815391    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c3ae00-3ded-496c-86d4-b4bc3c9917a9" containerName="revoke-keys"
Oct 30 01:11:40.815701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.815687    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c3ae00-3ded-496c-86d4-b4bc3c9917a9" containerName="revoke-keys"
Oct 30 01:11:40.831083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.830127    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"]
Oct 30 01:11:40.831083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.830191    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.846663 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.846635    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:40.873451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.865527    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:11:40.873451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.867446    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbqt\" (UniqueName: \"kubernetes.io/projected/19a012c4-ed97-4f0d-96f5-bed737924c73-kube-api-access-kgbqt\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.873451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.867507    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.873451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.867551    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.968670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.968635    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbqt\" (UniqueName: \"kubernetes.io/projected/19a012c4-ed97-4f0d-96f5-bed737924c73-kube-api-access-kgbqt\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.968864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.968688    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.968864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.968726    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.969437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.969409    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.971487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.971461    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:40.989266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:40.989209    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbqt\" (UniqueName: \"kubernetes.io/projected/19a012c4-ed97-4f0d-96f5-bed737924c73-kube-api-access-kgbqt\") pod \"maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:41.174250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:41.174198    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:41.492782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:41.492746    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"]
Oct 30 01:11:41.493105 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:11:41.492747    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a012c4_ed97_4f0d_96f5_bed737924c73.slice/crio-b5b3e126da305ab7d6e55a5d7606323a8e4a61f3ed41d70fc6bdb1b1d07af14b WatchSource:0}: Error finding container b5b3e126da305ab7d6e55a5d7606323a8e4a61f3ed41d70fc6bdb1b1d07af14b: Status 404 returned error can't find the container with id b5b3e126da305ab7d6e55a5d7606323a8e4a61f3ed41d70fc6bdb1b1d07af14b
Oct 30 01:11:41.554797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:41.554767    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z" event={"ID":"19a012c4-ed97-4f0d-96f5-bed737924c73","Type":"ContainerStarted","Data":"b5b3e126da305ab7d6e55a5d7606323a8e4a61f3ed41d70fc6bdb1b1d07af14b"}
Oct 30 01:11:42.095139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:42.095052    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:11:42.129371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:42.129337    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:42.564307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:42.564276    4576 generic.go:356] "Generic (PLEG): container finished" podID="19a012c4-ed97-4f0d-96f5-bed737924c73" containerID="f18669579f0064f68e1d91ca5a7c59d6ffaff0d24e32d307ee7e177913a5761b" exitCode=0
Oct 30 01:11:42.564704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:42.564348    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z" event={"ID":"19a012c4-ed97-4f0d-96f5-bed737924c73","Type":"ContainerDied","Data":"f18669579f0064f68e1d91ca5a7c59d6ffaff0d24e32d307ee7e177913a5761b"}
Oct 30 01:11:43.424895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:43.424843    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p" podUID="a2a41a95-372b-4cc6-99f4-0b1c94325a4f" containerName="main" containerID="cri-o://bd65f5acf0249bfbf779c47f96a7b2fddab283a3a9623076956b35bd05049709" gracePeriod=45
Oct 30 01:11:43.455209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:43.455173    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p" podUID="a2a41a95-372b-4cc6-99f4-0b1c94325a4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": dial tcp 10.133.0.103:8000: connect: connection refused"
Oct 30 01:11:43.632158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:43.626083    4576 generic.go:356] "Generic (PLEG): container finished" podID="a2a41a95-372b-4cc6-99f4-0b1c94325a4f" containerID="bd65f5acf0249bfbf779c47f96a7b2fddab283a3a9623076956b35bd05049709" exitCode=0
Oct 30 01:11:43.632158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:43.626394    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p" event={"ID":"a2a41a95-372b-4cc6-99f4-0b1c94325a4f","Type":"ContainerDied","Data":"bd65f5acf0249bfbf779c47f96a7b2fddab283a3a9623076956b35bd05049709"}
Oct 30 01:11:44.526269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.516566    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:44.546260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.544485    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:11:44.603753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.603721    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tls-certs\") pod \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") "
Oct 30 01:11:44.604097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.604065    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-home\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-home\") pod \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") "
Oct 30 01:11:44.718443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.686037    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-model-cache\") pod \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") "
Oct 30 01:11:44.718443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.686304    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-kube-api-access-4cn59\" (UniqueName: \"kubernetes.io/projected/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-kube-api-access-4cn59\") pod \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") "
Oct 30 01:11:44.718443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.686805    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-dshm\") pod \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") "
Oct 30 01:11:44.718443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.686974    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tmp-dir\") pod \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\" (UID: \"a2a41a95-372b-4cc6-99f4-0b1c94325a4f\") "
Oct 30 01:11:44.718443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.605139    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-home" pod "a2a41a95-372b-4cc6-99f4-0b1c94325a4f" (UID: "a2a41a95-372b-4cc6-99f4-0b1c94325a4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:44.718443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.688724    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tls-certs" pod "a2a41a95-372b-4cc6-99f4-0b1c94325a4f" (UID: "a2a41a95-372b-4cc6-99f4-0b1c94325a4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:44.718443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.709270    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tmp-dir" pod "a2a41a95-372b-4cc6-99f4-0b1c94325a4f" (UID: "a2a41a95-372b-4cc6-99f4-0b1c94325a4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:44.718443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.709573    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-model-cache" pod "a2a41a95-372b-4cc6-99f4-0b1c94325a4f" (UID: "a2a41a95-372b-4cc6-99f4-0b1c94325a4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:44.729963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.729897    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-dshm" pod "a2a41a95-372b-4cc6-99f4-0b1c94325a4f" (UID: "a2a41a95-372b-4cc6-99f4-0b1c94325a4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:44.747350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.747220    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:44.748123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.748083    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-kube-api-access-4cn59" pod "a2a41a95-372b-4cc6-99f4-0b1c94325a4f" (UID: "a2a41a95-372b-4cc6-99f4-0b1c94325a4f"). InnerVolumeSpecName "kube-api-access-4cn59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:44.754204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.754148    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z" event={"ID":"19a012c4-ed97-4f0d-96f5-bed737924c73","Type":"ContainerDied","Data":"b5b3e126da305ab7d6e55a5d7606323a8e4a61f3ed41d70fc6bdb1b1d07af14b"}
Oct 30 01:11:44.754372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.754276    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5b3e126da305ab7d6e55a5d7606323a8e4a61f3ed41d70fc6bdb1b1d07af14b"
Oct 30 01:11:44.754372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.754366    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"
Oct 30 01:11:44.756752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.756721    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p" event={"ID":"a2a41a95-372b-4cc6-99f4-0b1c94325a4f","Type":"ContainerDied","Data":"13e06e4b893c8f4bcbff7e60d2f059544705fdea3e78e6c5f2db4267ff65a581"}
Oct 30 01:11:44.756864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.756841    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"
Oct 30 01:11:44.757087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.757073    4576 scope.go:122] "RemoveContainer" containerID="bd65f5acf0249bfbf779c47f96a7b2fddab283a3a9623076956b35bd05049709"
Oct 30 01:11:44.810308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.810275    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a012c4-ed97-4f0d-96f5-bed737924c73-kube-api-access-kgbqt\" (UniqueName: \"kubernetes.io/projected/19a012c4-ed97-4f0d-96f5-bed737924c73-kube-api-access-kgbqt\") pod \"19a012c4-ed97-4f0d-96f5-bed737924c73\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") "
Oct 30 01:11:44.810700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.810673    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-xks-ca\") pod \"19a012c4-ed97-4f0d-96f5-bed737924c73\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") "
Oct 30 01:11:44.810976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.810956    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-openshift-ca\") pod \"19a012c4-ed97-4f0d-96f5-bed737924c73\" (UID: \"19a012c4-ed97-4f0d-96f5-bed737924c73\") "
Oct 30 01:11:44.811946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.811920    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:44.812072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.812058    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cn59\" (UniqueName: \"kubernetes.io/projected/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-kube-api-access-4cn59\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:44.812165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.812152    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:44.812294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.812281    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:44.812384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.812373    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:44.812479 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.812469    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a41a95-372b-4cc6-99f4-0b1c94325a4f-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:44.812916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.812894    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-openshift-ca" pod "19a012c4-ed97-4f0d-96f5-bed737924c73" (UID: "19a012c4-ed97-4f0d-96f5-bed737924c73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:44.822331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.822281    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a012c4-ed97-4f0d-96f5-bed737924c73-kube-api-access-kgbqt" pod "19a012c4-ed97-4f0d-96f5-bed737924c73" (UID: "19a012c4-ed97-4f0d-96f5-bed737924c73"). InnerVolumeSpecName "kube-api-access-kgbqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:44.824711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.824676    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-xks-ca" pod "19a012c4-ed97-4f0d-96f5-bed737924c73" (UID: "19a012c4-ed97-4f0d-96f5-bed737924c73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:44.834006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.833961    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"]
Oct 30 01:11:44.842066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.842010    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-49d9f1/test-model-49d9f1-kserve-7476f54b8f-k9q8p"]
Oct 30 01:11:44.914269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.914136    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgbqt\" (UniqueName: \"kubernetes.io/projected/19a012c4-ed97-4f0d-96f5-bed737924c73-kube-api-access-kgbqt\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:44.914467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.914279    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:44.914467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:44.914332    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a012c4-ed97-4f0d-96f5-bed737924c73-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:46.110377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:46.110343    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a41a95-372b-4cc6-99f4-0b1c94325a4f" path="/var/lib/kubelet/pods/a2a41a95-372b-4cc6-99f4-0b1c94325a4f/volumes"
Oct 30 01:11:46.265185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:46.265157    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:47.984250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:47.984208    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:48.383173 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:11:48.383094    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:11:48.939394 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:48.939360    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:49.467368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.467336    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"]
Oct 30 01:11:49.468097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.468080    4576 cpu_manager.go:414] "containerMap: removing container" podUID="19a012c4-ed97-4f0d-96f5-bed737924c73" containerName="revoke-keys"
Oct 30 01:11:49.468137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.468099    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a012c4-ed97-4f0d-96f5-bed737924c73" containerName="revoke-keys"
Oct 30 01:11:49.468137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.468118    4576 cpu_manager.go:414] "containerMap: removing container" podUID="a2a41a95-372b-4cc6-99f4-0b1c94325a4f" containerName="main"
Oct 30 01:11:49.468137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.468124    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a41a95-372b-4cc6-99f4-0b1c94325a4f" containerName="main"
Oct 30 01:11:49.468277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.468267    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a012c4-ed97-4f0d-96f5-bed737924c73" containerName="revoke-keys"
Oct 30 01:11:49.468335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.468280    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a41a95-372b-4cc6-99f4-0b1c94325a4f" containerName="main"
Oct 30 01:11:49.474422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.474395    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.489172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.489142    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"]
Oct 30 01:11:49.497580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.497555    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:49.497851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.497826    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:11:49.578623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.578587    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:49.591171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.591141    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmw74\" (UniqueName: \"kubernetes.io/projected/2c36dac6-0b58-4af3-856a-3f7cea301cde-kube-api-access-fmw74\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.591329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.591279    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.591478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.591458    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.693474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.693436    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmw74\" (UniqueName: \"kubernetes.io/projected/2c36dac6-0b58-4af3-856a-3f7cea301cde-kube-api-access-fmw74\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.693694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.693557    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.693762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.693728    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.694466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.694444    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.696260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.696218    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.714906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.714877    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmw74\" (UniqueName: \"kubernetes.io/projected/2c36dac6-0b58-4af3-856a-3f7cea301cde-kube-api-access-fmw74\") pod \"maas-api-revoke-sub-0f1e1fee-zzcqr\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:49.794661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:49.794584    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:50.096523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:50.096488    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"]
Oct 30 01:11:50.097518 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:11:50.097451    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c36dac6_0b58_4af3_856a_3f7cea301cde.slice/crio-d894feed2d1da835c134151f11fc04a9e3bd2988ba0363d57231cb2824fe705b WatchSource:0}: Error finding container d894feed2d1da835c134151f11fc04a9e3bd2988ba0363d57231cb2824fe705b: Status 404 returned error can't find the container with id d894feed2d1da835c134151f11fc04a9e3bd2988ba0363d57231cb2824fe705b
Oct 30 01:11:50.809068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:50.808974    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr" event={"ID":"2c36dac6-0b58-4af3-856a-3f7cea301cde","Type":"ContainerStarted","Data":"349ba32179b62e5eedea71a6634768ab1c5958f16333ddf5832639791e19b9fe"}
Oct 30 01:11:50.809068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:50.809041    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr" event={"ID":"2c36dac6-0b58-4af3-856a-3f7cea301cde","Type":"ContainerStarted","Data":"d894feed2d1da835c134151f11fc04a9e3bd2988ba0363d57231cb2824fe705b"}
Oct 30 01:11:50.859050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:50.858998    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr" podStartSLOduration=1.8589848 podStartE2EDuration="1.8589848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:50.854604768 +0000 UTC m=+1990.925928603" watchObservedRunningTime="2026-10-30 01:11:50.8589848 +0000 UTC m=+1990.930308633"
Oct 30 01:11:51.201919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:51.201883    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:51.506770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:51.506737    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:51.620079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:51.620048    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:51.818453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:51.818365    4576 generic.go:356] "Generic (PLEG): container finished" podID="2c36dac6-0b58-4af3-856a-3f7cea301cde" containerID="349ba32179b62e5eedea71a6634768ab1c5958f16333ddf5832639791e19b9fe" exitCode=0
Oct 30 01:11:51.818895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:51.818469    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr" event={"ID":"2c36dac6-0b58-4af3-856a-3f7cea301cde","Type":"ContainerDied","Data":"349ba32179b62e5eedea71a6634768ab1c5958f16333ddf5832639791e19b9fe"}
Oct 30 01:11:52.747398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:52.747348    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:53.401596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.401567    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:53.422016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.421984    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:53.548119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.548088    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-xks-ca\") pod \"2c36dac6-0b58-4af3-856a-3f7cea301cde\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") "
Oct 30 01:11:53.548494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.548152    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c36dac6-0b58-4af3-856a-3f7cea301cde-kube-api-access-fmw74\" (UniqueName: \"kubernetes.io/projected/2c36dac6-0b58-4af3-856a-3f7cea301cde-kube-api-access-fmw74\") pod \"2c36dac6-0b58-4af3-856a-3f7cea301cde\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") "
Oct 30 01:11:53.548494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.548294    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-openshift-ca\") pod \"2c36dac6-0b58-4af3-856a-3f7cea301cde\" (UID: \"2c36dac6-0b58-4af3-856a-3f7cea301cde\") "
Oct 30 01:11:53.548711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.548683    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-openshift-ca" pod "2c36dac6-0b58-4af3-856a-3f7cea301cde" (UID: "2c36dac6-0b58-4af3-856a-3f7cea301cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:53.549010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.548977    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:53.550534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.550509    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c36dac6-0b58-4af3-856a-3f7cea301cde-kube-api-access-fmw74" pod "2c36dac6-0b58-4af3-856a-3f7cea301cde" (UID: "2c36dac6-0b58-4af3-856a-3f7cea301cde"). InnerVolumeSpecName "kube-api-access-fmw74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:53.550629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.550551    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-xks-ca" pod "2c36dac6-0b58-4af3-856a-3f7cea301cde" (UID: "2c36dac6-0b58-4af3-856a-3f7cea301cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:53.622161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.622067    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:53.635171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.635142    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"]
Oct 30 01:11:53.639300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.639268    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49d9f1-cfh6z"]
Oct 30 01:11:53.650379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.650343    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c36dac6-0b58-4af3-856a-3f7cea301cde-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:53.650379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.650379    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmw74\" (UniqueName: \"kubernetes.io/projected/2c36dac6-0b58-4af3-856a-3f7cea301cde-kube-api-access-fmw74\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:11:53.747672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.746816    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:53.835250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.835198    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr" event={"ID":"2c36dac6-0b58-4af3-856a-3f7cea301cde","Type":"ContainerDied","Data":"d894feed2d1da835c134151f11fc04a9e3bd2988ba0363d57231cb2824fe705b"}
Oct 30 01:11:53.835250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.835224    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"
Oct 30 01:11:53.835460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:53.835266    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d894feed2d1da835c134151f11fc04a9e3bd2988ba0363d57231cb2824fe705b"
Oct 30 01:11:54.112085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:54.112050    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a012c4-ed97-4f0d-96f5-bed737924c73" path="/var/lib/kubelet/pods/19a012c4-ed97-4f0d-96f5-bed737924c73/volumes"
Oct 30 01:11:54.136584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:54.136549    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:55.515953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:55.515913    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:55.626941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:55.626896    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:56.910304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:56.910268    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:57.046811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:57.046720    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:58.638683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:58.638623    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:11:59.932816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:11:59.932777    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:00.299159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.299127    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:00.436120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.436080    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:00.617855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.617627    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:00.635380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.635202    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"]
Oct 30 01:12:00.635918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.635839    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy" containerID="cri-o://c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855" gracePeriod=30
Oct 30 01:12:00.736688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.736551    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:00.878393 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.878293    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:00.978830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.978767    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"]
Oct 30 01:12:00.982887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:00.982861    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b483fd-l2k28"]
Oct 30 01:12:01.053844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.053800    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:01.190337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.190304    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"]
Oct 30 01:12:01.191939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.191901    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2c36dac6-0b58-4af3-856a-3f7cea301cde" containerName="revoke-keys"
Oct 30 01:12:01.191939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.191931    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c36dac6-0b58-4af3-856a-3f7cea301cde" containerName="revoke-keys"
Oct 30 01:12:01.192605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.192171    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c36dac6-0b58-4af3-856a-3f7cea301cde" containerName="revoke-keys"
Oct 30 01:12:01.202572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.197938    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"]
Oct 30 01:12:01.202572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.198086    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.220292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.216870    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:01.220292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.216897    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:12:01.306756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.306723    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.306942 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.306774    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.306990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.306972    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmld\" (UniqueName: \"kubernetes.io/projected/55a6012a-c71a-4df8-b7df-3491b6527a95-kube-api-access-8rmld\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.400448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.400416    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:01.408851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.408824    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmld\" (UniqueName: \"kubernetes.io/projected/55a6012a-c71a-4df8-b7df-3491b6527a95-kube-api-access-8rmld\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.409003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.408984    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.409047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.409024    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.409953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.409926    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.411764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.411742    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.430908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.430881    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmld\" (UniqueName: \"kubernetes.io/projected/55a6012a-c71a-4df8-b7df-3491b6527a95-kube-api-access-8rmld\") pod \"maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.524019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.523939    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:01.832106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.832079    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"]
Oct 30 01:12:01.833641 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:12:01.833612    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a6012a_c71a_4df8_b7df_3491b6527a95.slice/crio-cdb375e8c8246cefe4ea3a0b0c303758d296c66c1c0c7e6d923cd8d2391a2eaf WatchSource:0}: Error finding container cdb375e8c8246cefe4ea3a0b0c303758d296c66c1c0c7e6d923cd8d2391a2eaf: Status 404 returned error can't find the container with id cdb375e8c8246cefe4ea3a0b0c303758d296c66c1c0c7e6d923cd8d2391a2eaf
Oct 30 01:12:01.909716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:01.909686    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj" event={"ID":"55a6012a-c71a-4df8-b7df-3491b6527a95","Type":"ContainerStarted","Data":"cdb375e8c8246cefe4ea3a0b0c303758d296c66c1c0c7e6d923cd8d2391a2eaf"}
Oct 30 01:12:02.113278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:02.113217    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7005b725-d30f-4079-9eda-3eb35e70f796" path="/var/lib/kubelet/pods/7005b725-d30f-4079-9eda-3eb35e70f796/volumes"
Oct 30 01:12:02.919104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:02.919059    4576 generic.go:356] "Generic (PLEG): container finished" podID="55a6012a-c71a-4df8-b7df-3491b6527a95" containerID="a57a61620b4dfe51f6e59b35f276df6ba884a1052ee29a7a6c921787335240d0" exitCode=0
Oct 30 01:12:02.919389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:02.919164    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj" event={"ID":"55a6012a-c71a-4df8-b7df-3491b6527a95","Type":"ContainerDied","Data":"a57a61620b4dfe51f6e59b35f276df6ba884a1052ee29a7a6c921787335240d0"}
Oct 30 01:12:03.032306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:03.032160    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:03.149108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:03.149070    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:03.256784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:03.256743    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:03.618917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:03.618882    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"]
Oct 30 01:12:03.619121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:03.619095    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy" containerID="cri-o://fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4" gracePeriod=30
Oct 30 01:12:04.124168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.124134    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:04.271389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.256873    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-xks-ca\") pod \"55a6012a-c71a-4df8-b7df-3491b6527a95\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") "
Oct 30 01:12:04.285666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.285623    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-openshift-ca\") pod \"55a6012a-c71a-4df8-b7df-3491b6527a95\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") "
Oct 30 01:12:04.286489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.286467    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a6012a-c71a-4df8-b7df-3491b6527a95-kube-api-access-8rmld\" (UniqueName: \"kubernetes.io/projected/55a6012a-c71a-4df8-b7df-3491b6527a95-kube-api-access-8rmld\") pod \"55a6012a-c71a-4df8-b7df-3491b6527a95\" (UID: \"55a6012a-c71a-4df8-b7df-3491b6527a95\") "
Oct 30 01:12:04.287781 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.266997    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-xks-ca" pod "55a6012a-c71a-4df8-b7df-3491b6527a95" (UID: "55a6012a-c71a-4df8-b7df-3491b6527a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:04.287970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.286224    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-openshift-ca" pod "55a6012a-c71a-4df8-b7df-3491b6527a95" (UID: "55a6012a-c71a-4df8-b7df-3491b6527a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:04.294639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.294608    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a6012a-c71a-4df8-b7df-3491b6527a95-kube-api-access-8rmld" pod "55a6012a-c71a-4df8-b7df-3491b6527a95" (UID: "55a6012a-c71a-4df8-b7df-3491b6527a95"). InnerVolumeSpecName "kube-api-access-8rmld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:04.389895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.389858    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:04.389895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.389895    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55a6012a-c71a-4df8-b7df-3491b6527a95-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:04.390135 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.389908    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rmld\" (UniqueName: \"kubernetes.io/projected/55a6012a-c71a-4df8-b7df-3491b6527a95-kube-api-access-8rmld\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:04.879934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.879897    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:04.944282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.943294    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj" event={"ID":"55a6012a-c71a-4df8-b7df-3491b6527a95","Type":"ContainerDied","Data":"cdb375e8c8246cefe4ea3a0b0c303758d296c66c1c0c7e6d923cd8d2391a2eaf"}
Oct 30 01:12:04.944282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.943335    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"
Oct 30 01:12:04.944282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:04.943362    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdb375e8c8246cefe4ea3a0b0c303758d296c66c1c0c7e6d923cd8d2391a2eaf"
Oct 30 01:12:05.938985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:05.938957    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:12:05.952730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:05.952699    4576 generic.go:356] "Generic (PLEG): container finished" podID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerID="c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855" exitCode=0
Oct 30 01:12:05.952907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:05.952752    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"
Oct 30 01:12:05.952907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:05.952785    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" event={"ID":"0822bea0-f14a-408f-8dd1-6dfa89de15b1","Type":"ContainerDied","Data":"c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855"}
Oct 30 01:12:05.952907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:05.952840    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl" event={"ID":"0822bea0-f14a-408f-8dd1-6dfa89de15b1","Type":"ContainerDied","Data":"dde93ca2273a4a00fbfb69f159608447e8a349f54b366b8c9d1725dd1fee66ce"}
Oct 30 01:12:05.952907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:05.952863    4576 scope.go:122] "RemoveContainer" containerID="c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855"
Oct 30 01:12:05.964439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:05.964420    4576 scope.go:122] "RemoveContainer" containerID="c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855"
Oct 30 01:12:05.964747 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:12:05.964726    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855\": container with ID starting with c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855 not found: ID does not exist" containerID="c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855"
Oct 30 01:12:05.964843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:05.964756    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855"} err="failed to get container status \"c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855\": rpc error: code = NotFound desc = could not find container \"c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855\": container with ID starting with c61aef021bdfbcb4179bb45ae32eabf1b419c6084095f1d11467859628c3b855 not found: ID does not exist"
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.011323    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-podinfo\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.011380    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-certs\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.011434    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-credential-socket\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.011527    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-data\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.011579    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-socket\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.011686    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-token\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-token\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.011855    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-envoy\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.011910    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istiod-ca-cert\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.012071    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-kube-api-access-t2l7h\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-kube-api-access-t2l7h\") pod \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\" (UID: \"0822bea0-f14a-408f-8dd1-6dfa89de15b1\") "
Oct 30 01:12:06.012979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.012946    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-data" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:06.015692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.015665    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-token" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:06.015908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.015882    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-credential-socket" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:06.015908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.015899    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-certs" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:06.016050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.015990    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-socket" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:06.016269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.016209    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istiod-ca-cert" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:06.017020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.016904    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-podinfo" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:06.017020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.016988    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-kube-api-access-t2l7h" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "kube-api-access-t2l7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:06.018189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.018164    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-envoy" pod "0822bea0-f14a-408f-8dd1-6dfa89de15b1" (UID: "0822bea0-f14a-408f-8dd1-6dfa89de15b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:06.113873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113842    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.113873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113870    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.113873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113879    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.114158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113888    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.114158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113897    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.114158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113905    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.114158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113912    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.114158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113920    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0822bea0-f14a-408f-8dd1-6dfa89de15b1-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.114158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.113928    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2l7h\" (UniqueName: \"kubernetes.io/projected/0822bea0-f14a-408f-8dd1-6dfa89de15b1-kube-api-access-t2l7h\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:06.296938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.296856    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"]
Oct 30 01:12:06.300560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.300534    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49d9f1-openshift-default-7d8f8bf56f-cf2kl"]
Oct 30 01:12:06.564543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:06.564434    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:07.407407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:07.407369    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:07.527556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:07.527522    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:07.693584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:07.693548    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"]
Oct 30 01:12:07.693870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:07.693840    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy" containerID="cri-o://f5c435eb23cfda2b16f84e9720920a2d30b1e05d3da2d278037f3e1d0483a6ea" gracePeriod=30
Oct 30 01:12:07.748858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:07.748826    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:08.111310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.111218    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" path="/var/lib/kubelet/pods/0822bea0-f14a-408f-8dd1-6dfa89de15b1/volumes"
Oct 30 01:12:08.565112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.565062    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:08.914253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.914200    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 01:12:08.984968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.984919    4576 generic.go:356] "Generic (PLEG): container finished" podID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerID="fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4" exitCode=0
Oct 30 01:12:08.985171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.985009    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" event={"ID":"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4","Type":"ContainerDied","Data":"fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4"}
Oct 30 01:12:08.985171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.985058    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l" event={"ID":"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4","Type":"ContainerDied","Data":"3d90b7da09fcb1bc5858127e11c8484eebfb4f1ee4a5b7364a3e338c137a40ff"}
Oct 30 01:12:08.985171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.985091    4576 scope.go:122] "RemoveContainer" containerID="fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4"
Oct 30 01:12:08.985562 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.985544    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"
Oct 30 01:12:08.999459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.999431    4576 scope.go:122] "RemoveContainer" containerID="fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4"
Oct 30 01:12:08.999810 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:12:08.999773    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4\": container with ID starting with fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4 not found: ID does not exist" containerID="fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4"
Oct 30 01:12:08.999947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:08.999817    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4"} err="failed to get container status \"fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4\": rpc error: code = NotFound desc = could not find container \"fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4\": container with ID starting with fa363fd355fe687501ae942e3ca6038e70a27965791f3681cb8e3f7fcd52a6b4 not found: ID does not exist"
Oct 30 01:12:09.067109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067073    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istiod-ca-cert\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067166    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-certs\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067226    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-kube-api-access-dv9df\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-kube-api-access-dv9df\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067313    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-envoy\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067405    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-credential-socket\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067488    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-socket\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067527    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-data\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067591    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-podinfo\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067594    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-certs" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:09.067758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067730    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-token\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-token\") pod \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\" (UID: \"3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4\") "
Oct 30 01:12:09.067835 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.067813    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-socket" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:09.068315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.068289    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istiod-ca-cert" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:09.068412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.068374    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-data" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:09.068551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.068516    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.068551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.068545    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.068701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.068559    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.068701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.068572    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.068701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.068635    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-credential-socket" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:09.070270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.070211    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-token" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:09.070386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.070363    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-envoy" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:09.070636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.070615    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-kube-api-access-dv9df" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "kube-api-access-dv9df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:09.070772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.070753    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-podinfo" pod "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" (UID: "3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:09.170370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.170290    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.170370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.170317    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.170370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.170328    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.170370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.170336    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv9df\" (UniqueName: \"kubernetes.io/projected/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-kube-api-access-dv9df\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.170370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.170345    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:09.339530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.339495    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"]
Oct 30 01:12:09.343088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.343030    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-vgn6l"]
Oct 30 01:12:09.502010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.501976    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"]
Oct 30 01:12:09.502724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502700    4576 cpu_manager.go:414] "containerMap: removing container" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy"
Oct 30 01:12:09.502724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502720    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy"
Oct 30 01:12:09.502902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502751    4576 cpu_manager.go:414] "containerMap: removing container" podUID="55a6012a-c71a-4df8-b7df-3491b6527a95" containerName="revoke-keys"
Oct 30 01:12:09.502902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502756    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a6012a-c71a-4df8-b7df-3491b6527a95" containerName="revoke-keys"
Oct 30 01:12:09.502902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502774    4576 cpu_manager.go:414] "containerMap: removing container" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy"
Oct 30 01:12:09.502902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502778    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy"
Oct 30 01:12:09.502902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502879    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a6012a-c71a-4df8-b7df-3491b6527a95" containerName="revoke-keys"
Oct 30 01:12:09.502902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502892    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" containerName="istio-proxy"
Oct 30 01:12:09.503079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.502908    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="0822bea0-f14a-408f-8dd1-6dfa89de15b1" containerName="istio-proxy"
Oct 30 01:12:09.507900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.507880    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.524514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.524476    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:12:09.524666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.524552    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:09.527830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.527802    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"]
Oct 30 01:12:09.678085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.678050    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9djp\" (UniqueName: \"kubernetes.io/projected/4e87c176-6d2e-47cd-a5c5-73075381d8f7-kube-api-access-l9djp\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.678505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.678097    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.678505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.678120    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.780157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.780082    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9djp\" (UniqueName: \"kubernetes.io/projected/4e87c176-6d2e-47cd-a5c5-73075381d8f7-kube-api-access-l9djp\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.780157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.780127    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.780157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.780152    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.781441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.781411    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.795214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.795182    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.800629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.800602    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9djp\" (UniqueName: \"kubernetes.io/projected/4e87c176-6d2e-47cd-a5c5-73075381d8f7-kube-api-access-l9djp\") pod \"maas-api-revoke-sub-8da3a029-lcb9m\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:09.823576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:09.823556    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:10.110362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:10.110220    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4" path="/var/lib/kubelet/pods/3d3049ef-0be4-4a1b-8c6d-d05d5e81b1b4/volumes"
Oct 30 01:12:10.131645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:10.131612    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"]
Oct 30 01:12:10.132866 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:12:10.132838    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e87c176_6d2e_47cd_a5c5_73075381d8f7.slice/crio-bf1e2c351a755bda6a3ba81a950db95a3cc832c93633559e4ed7c0aed6cb2fbe WatchSource:0}: Error finding container bf1e2c351a755bda6a3ba81a950db95a3cc832c93633559e4ed7c0aed6cb2fbe: Status 404 returned error can't find the container with id bf1e2c351a755bda6a3ba81a950db95a3cc832c93633559e4ed7c0aed6cb2fbe
Oct 30 01:12:10.202305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:10.202270    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:10.843482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:10.843394    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"]
Oct 30 01:12:10.904642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:10.904607    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:11.029302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.029267    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m" event={"ID":"4e87c176-6d2e-47cd-a5c5-73075381d8f7","Type":"ContainerStarted","Data":"b046da7d69ebd29ae4412bd6f9d0f01fcbbc536a70148a5cda3d1ed8730bbca7"}
Oct 30 01:12:11.029594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.029575    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m" event={"ID":"4e87c176-6d2e-47cd-a5c5-73075381d8f7","Type":"ContainerStarted","Data":"bf1e2c351a755bda6a3ba81a950db95a3cc832c93633559e4ed7c0aed6cb2fbe"}
Oct 30 01:12:11.070925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.070880    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"]
Oct 30 01:12:11.079434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.079405    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.090389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.087369    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m" podStartSLOduration=2.087355057 podStartE2EDuration="2.087355057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:11.074521234 +0000 UTC m=+2011.145845070" watchObservedRunningTime="2026-10-30 01:12:11.087355057 +0000 UTC m=+2011.158678928"
Oct 30 01:12:11.090389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.089814    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"]
Oct 30 01:12:11.207865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.207834    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.208784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.208752    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.208917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.208819    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pg6v\" (UniqueName: \"kubernetes.io/projected/a1868cc4-b89d-494c-9c17-057f05981ea6-kube-api-access-5pg6v\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.310713 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.310683    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.310813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.310748    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pg6v\" (UniqueName: \"kubernetes.io/projected/a1868cc4-b89d-494c-9c17-057f05981ea6-kube-api-access-5pg6v\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.310862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.310845    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.311403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.311381    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.313520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.313501    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.332276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.332222    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pg6v\" (UniqueName: \"kubernetes.io/projected/a1868cc4-b89d-494c-9c17-057f05981ea6-kube-api-access-5pg6v\") pod \"maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.434474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.434387    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:11.738127 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:12:11.738094    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1868cc4_b89d_494c_9c17_057f05981ea6.slice/crio-7745f41fc2e3855e1620b213a46d59ce508a49f1a63d3ef51ee2046772561f10 WatchSource:0}: Error finding container 7745f41fc2e3855e1620b213a46d59ce508a49f1a63d3ef51ee2046772561f10: Status 404 returned error can't find the container with id 7745f41fc2e3855e1620b213a46d59ce508a49f1a63d3ef51ee2046772561f10
Oct 30 01:12:11.738324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.738306    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"]
Oct 30 01:12:11.949851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.949821    4576 patch_prober.go:29] interesting pod/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:12:11.950251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.949871    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:12:11.991118 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:11.991037    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"]
Oct 30 01:12:12.040904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:12.040863    4576 generic.go:356] "Generic (PLEG): container finished" podID="4e87c176-6d2e-47cd-a5c5-73075381d8f7" containerID="b046da7d69ebd29ae4412bd6f9d0f01fcbbc536a70148a5cda3d1ed8730bbca7" exitCode=0
Oct 30 01:12:12.041120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:12.040911    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m" event={"ID":"4e87c176-6d2e-47cd-a5c5-73075381d8f7","Type":"ContainerDied","Data":"b046da7d69ebd29ae4412bd6f9d0f01fcbbc536a70148a5cda3d1ed8730bbca7"}
Oct 30 01:12:12.043227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:12.043192    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct" event={"ID":"a1868cc4-b89d-494c-9c17-057f05981ea6","Type":"ContainerStarted","Data":"7745f41fc2e3855e1620b213a46d59ce508a49f1a63d3ef51ee2046772561f10"}
Oct 30 01:12:12.116186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:12.116151    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:12.248679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:12.248597    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:12.357686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:12.357652    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:13.053193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.053094    4576 generic.go:356] "Generic (PLEG): container finished" podID="a1868cc4-b89d-494c-9c17-057f05981ea6" containerID="c9ebb32708f2fb9ec3baac5eb99bc2d994fc04ea1fcafca6c1d9b77fae37ddc2" exitCode=0
Oct 30 01:12:13.053636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.053183    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct" event={"ID":"a1868cc4-b89d-494c-9c17-057f05981ea6","Type":"ContainerDied","Data":"c9ebb32708f2fb9ec3baac5eb99bc2d994fc04ea1fcafca6c1d9b77fae37ddc2"}
Oct 30 01:12:13.055041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.055017    4576 generic.go:356] "Generic (PLEG): container finished" podID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerID="f5c435eb23cfda2b16f84e9720920a2d30b1e05d3da2d278037f3e1d0483a6ea" exitCode=0
Oct 30 01:12:13.055171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.055083    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" event={"ID":"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e","Type":"ContainerDied","Data":"f5c435eb23cfda2b16f84e9720920a2d30b1e05d3da2d278037f3e1d0483a6ea"}
Oct 30 01:12:13.360762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.360730    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:13.399152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.399113    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-xks-ca\") pod \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") "
Oct 30 01:12:13.399364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.399264    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-openshift-ca\") pod \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") "
Oct 30 01:12:13.399446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.399364    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e87c176-6d2e-47cd-a5c5-73075381d8f7-kube-api-access-l9djp\" (UniqueName: \"kubernetes.io/projected/4e87c176-6d2e-47cd-a5c5-73075381d8f7-kube-api-access-l9djp\") pod \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\" (UID: \"4e87c176-6d2e-47cd-a5c5-73075381d8f7\") "
Oct 30 01:12:13.400274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.399661    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-openshift-ca" pod "4e87c176-6d2e-47cd-a5c5-73075381d8f7" (UID: "4e87c176-6d2e-47cd-a5c5-73075381d8f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:13.400274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.400173    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.403456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.403423    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e87c176-6d2e-47cd-a5c5-73075381d8f7-kube-api-access-l9djp" pod "4e87c176-6d2e-47cd-a5c5-73075381d8f7" (UID: "4e87c176-6d2e-47cd-a5c5-73075381d8f7"). InnerVolumeSpecName "kube-api-access-l9djp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:13.404092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.404053    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-xks-ca" pod "4e87c176-6d2e-47cd-a5c5-73075381d8f7" (UID: "4e87c176-6d2e-47cd-a5c5-73075381d8f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:13.466818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.466788    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:12:13.500911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.500883    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-certs\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.500970    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-token\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-token\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.500990    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-envoy\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501042    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-data\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501062    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-podinfo\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501122    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istiod-ca-cert\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501195    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-certs" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:13.501423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501244    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-kube-api-access-9plmf\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-kube-api-access-9plmf\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501289    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-credential-socket\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501360    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-socket\") pod \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\" (UID: \"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e\") "
Oct 30 01:12:13.501804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501726    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-data" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:13.501908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501828    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-credential-socket" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:13.501985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501957    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-socket" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:13.501985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.501965    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istiod-ca-cert" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:13.502145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.502130    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.502261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.502151    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.502261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.502168    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.502261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.502181    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e87c176-6d2e-47cd-a5c5-73075381d8f7-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.502261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.502193    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.502261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.502205    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9djp\" (UniqueName: \"kubernetes.io/projected/4e87c176-6d2e-47cd-a5c5-73075381d8f7-kube-api-access-l9djp\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.502261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.502218    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.503910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.503879    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-envoy" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:13.503986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.503965    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-podinfo" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:12:13.504023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.503994    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-token" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:13.504505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.504487    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-kube-api-access-9plmf" pod "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" (UID: "e96b3f6f-3e9b-495f-bde2-1c83f9b6753e"). InnerVolumeSpecName "kube-api-access-9plmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:13.603302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.603176    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.603302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.603208    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.603302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.603220    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:13.603302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:13.603250    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9plmf\" (UniqueName: \"kubernetes.io/projected/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e-kube-api-access-9plmf\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:14.154803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.154770    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"
Oct 30 01:12:14.181491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.181448    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m" event={"ID":"4e87c176-6d2e-47cd-a5c5-73075381d8f7","Type":"ContainerDied","Data":"bf1e2c351a755bda6a3ba81a950db95a3cc832c93633559e4ed7c0aed6cb2fbe"}
Oct 30 01:12:14.181661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.181522    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf1e2c351a755bda6a3ba81a950db95a3cc832c93633559e4ed7c0aed6cb2fbe"
Oct 30 01:12:14.217292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.215328    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"
Oct 30 01:12:14.225016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.224986    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp" event={"ID":"e96b3f6f-3e9b-495f-bde2-1c83f9b6753e","Type":"ContainerDied","Data":"c2032500a8f1f2f30ae5e84b43465abdc12d1bd9a984807b9f0f2c9ad8005ccb"}
Oct 30 01:12:14.225270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.225186    4576 scope.go:122] "RemoveContainer" containerID="f5c435eb23cfda2b16f84e9720920a2d30b1e05d3da2d278037f3e1d0483a6ea"
Oct 30 01:12:14.312277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.312216    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"]
Oct 30 01:12:14.327488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.327456    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b483fd-openshift-default-64bb6d775-5sgwp"]
Oct 30 01:12:14.437142 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.437116    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:14.471784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.471750    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:14.539225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.539190    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-xks-ca\") pod \"a1868cc4-b89d-494c-9c17-057f05981ea6\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") "
Oct 30 01:12:14.539451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.539425    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1868cc4-b89d-494c-9c17-057f05981ea6-kube-api-access-5pg6v\" (UniqueName: \"kubernetes.io/projected/a1868cc4-b89d-494c-9c17-057f05981ea6-kube-api-access-5pg6v\") pod \"a1868cc4-b89d-494c-9c17-057f05981ea6\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") "
Oct 30 01:12:14.539591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.539573    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-openshift-ca\") pod \"a1868cc4-b89d-494c-9c17-057f05981ea6\" (UID: \"a1868cc4-b89d-494c-9c17-057f05981ea6\") "
Oct 30 01:12:14.539938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.539910    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-openshift-ca" pod "a1868cc4-b89d-494c-9c17-057f05981ea6" (UID: "a1868cc4-b89d-494c-9c17-057f05981ea6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:14.540641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.540620    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:14.541794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.541770    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-xks-ca" pod "a1868cc4-b89d-494c-9c17-057f05981ea6" (UID: "a1868cc4-b89d-494c-9c17-057f05981ea6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:14.542587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.542561    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1868cc4-b89d-494c-9c17-057f05981ea6-kube-api-access-5pg6v" pod "a1868cc4-b89d-494c-9c17-057f05981ea6" (UID: "a1868cc4-b89d-494c-9c17-057f05981ea6"). InnerVolumeSpecName "kube-api-access-5pg6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:14.592823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.592788    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:14.641878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.641838    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pg6v\" (UniqueName: \"kubernetes.io/projected/a1868cc4-b89d-494c-9c17-057f05981ea6-kube-api-access-5pg6v\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:14.641878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:14.641865    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1868cc4-b89d-494c-9c17-057f05981ea6-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:15.234432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:15.234399    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"
Oct 30 01:12:15.234872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:15.234431    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct" event={"ID":"a1868cc4-b89d-494c-9c17-057f05981ea6","Type":"ContainerDied","Data":"7745f41fc2e3855e1620b213a46d59ce508a49f1a63d3ef51ee2046772561f10"}
Oct 30 01:12:15.234872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:15.234509    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7745f41fc2e3855e1620b213a46d59ce508a49f1a63d3ef51ee2046772561f10"
Oct 30 01:12:16.109821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:16.109781    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" path="/var/lib/kubelet/pods/e96b3f6f-3e9b-495f-bde2-1c83f9b6753e/volumes"
Oct 30 01:12:16.220494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:16.220459    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:16.991542 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:16.991507    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2" podUID="5c232f16-6971-4881-89a0-63a798665351" containerName="payload-pre-processing" containerID="cri-o://b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4" gracePeriod=25
Oct 30 01:12:17.773395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:17.773353    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:18.384047 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:12:18.384012    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:12:19.460098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:19.460066    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:20.315838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:20.315798    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2" podUID="5c232f16-6971-4881-89a0-63a798665351" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:12:21.050445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:21.050408    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:21.696666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:21.696564    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"]
Oct 30 01:12:21.911270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:21.908742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:22.237134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237104    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"]
Oct 30 01:12:22.237785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237769    4576 cpu_manager.go:414] "containerMap: removing container" podUID="4e87c176-6d2e-47cd-a5c5-73075381d8f7" containerName="revoke-keys"
Oct 30 01:12:22.237830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237787    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e87c176-6d2e-47cd-a5c5-73075381d8f7" containerName="revoke-keys"
Oct 30 01:12:22.237830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237806    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy"
Oct 30 01:12:22.237830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237811    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy"
Oct 30 01:12:22.237830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237831    4576 cpu_manager.go:414] "containerMap: removing container" podUID="a1868cc4-b89d-494c-9c17-057f05981ea6" containerName="revoke-keys"
Oct 30 01:12:22.237965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237836    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1868cc4-b89d-494c-9c17-057f05981ea6" containerName="revoke-keys"
Oct 30 01:12:22.237965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237941    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e87c176-6d2e-47cd-a5c5-73075381d8f7" containerName="revoke-keys"
Oct 30 01:12:22.237965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237950    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96b3f6f-3e9b-495f-bde2-1c83f9b6753e" containerName="istio-proxy"
Oct 30 01:12:22.237965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.237959    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1868cc4-b89d-494c-9c17-057f05981ea6" containerName="revoke-keys"
Oct 30 01:12:22.242823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.242808    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.260600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.260574    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"]
Oct 30 01:12:22.269729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.269702    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:22.269897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.269878    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:12:22.362443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.362403    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbrd\" (UniqueName: \"kubernetes.io/projected/d7f15448-71f1-4811-bd35-2c224f55d7eb-kube-api-access-8wbrd\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.362707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.362688    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.362759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.362724    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.464006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.463963    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbrd\" (UniqueName: \"kubernetes.io/projected/d7f15448-71f1-4811-bd35-2c224f55d7eb-kube-api-access-8wbrd\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.464211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.464179    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.464211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.464209    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.464963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.464943    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.466817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.466790    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.485182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.485150    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbrd\" (UniqueName: \"kubernetes.io/projected/d7f15448-71f1-4811-bd35-2c224f55d7eb-kube-api-access-8wbrd\") pod \"maas-api-revoke-sub-dfd8c650-m6k7c\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.557743 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.557671    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:22.738067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.736824    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:22.861848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.861754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:22.882847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:22.882813    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"]
Oct 30 01:12:22.884178 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:12:22.884149    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f15448_71f1_4811_bd35_2c224f55d7eb.slice/crio-710ebb87d9666ce63d164c0e444b7de76e877c382b401a0ae618ab26cef206e7 WatchSource:0}: Error finding container 710ebb87d9666ce63d164c0e444b7de76e877c382b401a0ae618ab26cef206e7: Status 404 returned error can't find the container with id 710ebb87d9666ce63d164c0e444b7de76e877c382b401a0ae618ab26cef206e7
Oct 30 01:12:23.307276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:23.302707    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c" event={"ID":"d7f15448-71f1-4811-bd35-2c224f55d7eb","Type":"ContainerStarted","Data":"6c97fe38217c1d8ebe6da2bab1303a5ddec09cfe85810d809f38e4e65892b7a4"}
Oct 30 01:12:23.307276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:23.302803    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c" event={"ID":"d7f15448-71f1-4811-bd35-2c224f55d7eb","Type":"ContainerStarted","Data":"710ebb87d9666ce63d164c0e444b7de76e877c382b401a0ae618ab26cef206e7"}
Oct 30 01:12:23.365332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:23.365274    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c" podStartSLOduration=1.3652594439999999 podStartE2EDuration="1.365259444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:23.361515066 +0000 UTC m=+2023.432838896" watchObservedRunningTime="2026-10-30 01:12:23.365259444 +0000 UTC m=+2023.436583278"
Oct 30 01:12:23.703507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:23.703474    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:23.813717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:23.813677    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:24.199646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:24.199608    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:24.314007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:24.313973    4576 generic.go:356] "Generic (PLEG): container finished" podID="d7f15448-71f1-4811-bd35-2c224f55d7eb" containerID="6c97fe38217c1d8ebe6da2bab1303a5ddec09cfe85810d809f38e4e65892b7a4" exitCode=0
Oct 30 01:12:24.314599 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:24.314061    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c" event={"ID":"d7f15448-71f1-4811-bd35-2c224f55d7eb","Type":"ContainerDied","Data":"6c97fe38217c1d8ebe6da2bab1303a5ddec09cfe85810d809f38e4e65892b7a4"}
Oct 30 01:12:24.895899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:24.895862    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:25.008623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.008581    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:25.021123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.021084    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"]
Oct 30 01:12:25.035166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.035128    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e8ad1-85gct"]
Oct 30 01:12:25.195088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.195058    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:25.484818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.484796    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:25.502325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.502300    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:25.617846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.617812    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-xks-ca\") pod \"d7f15448-71f1-4811-bd35-2c224f55d7eb\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") "
Oct 30 01:12:25.618016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.617909    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-openshift-ca\") pod \"d7f15448-71f1-4811-bd35-2c224f55d7eb\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") "
Oct 30 01:12:25.618016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.617942    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f15448-71f1-4811-bd35-2c224f55d7eb-kube-api-access-8wbrd\" (UniqueName: \"kubernetes.io/projected/d7f15448-71f1-4811-bd35-2c224f55d7eb-kube-api-access-8wbrd\") pod \"d7f15448-71f1-4811-bd35-2c224f55d7eb\" (UID: \"d7f15448-71f1-4811-bd35-2c224f55d7eb\") "
Oct 30 01:12:25.618316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.618274    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-openshift-ca" pod "d7f15448-71f1-4811-bd35-2c224f55d7eb" (UID: "d7f15448-71f1-4811-bd35-2c224f55d7eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:25.618715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.618693    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:25.620296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.620266    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-xks-ca" pod "d7f15448-71f1-4811-bd35-2c224f55d7eb" (UID: "d7f15448-71f1-4811-bd35-2c224f55d7eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:25.620296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.620271    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f15448-71f1-4811-bd35-2c224f55d7eb-kube-api-access-8wbrd" pod "d7f15448-71f1-4811-bd35-2c224f55d7eb" (UID: "d7f15448-71f1-4811-bd35-2c224f55d7eb"). InnerVolumeSpecName "kube-api-access-8wbrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:25.720839 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.720755    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbrd\" (UniqueName: \"kubernetes.io/projected/d7f15448-71f1-4811-bd35-2c224f55d7eb-kube-api-access-8wbrd\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:25.720839 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.720788    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f15448-71f1-4811-bd35-2c224f55d7eb-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:25.883773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:25.883732    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr" podUID="c4587c86-fe98-40d7-8473-30f6fb0f79dd" containerName="main" containerID="cri-o://25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b" gracePeriod=45
Oct 30 01:12:26.027358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.027320    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr" podUID="c4587c86-fe98-40d7-8473-30f6fb0f79dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/ready\": dial tcp 10.133.0.104:8000: connect: connection refused"
Oct 30 01:12:26.115159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.115117    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1868cc4-b89d-494c-9c17-057f05981ea6" path="/var/lib/kubelet/pods/a1868cc4-b89d-494c-9c17-057f05981ea6/volumes"
Oct 30 01:12:26.143460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.143434    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:12:26.226036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.225997    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tmp-dir\") pod \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") "
Oct 30 01:12:26.226276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.226089    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-home\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-home\") pod \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") "
Oct 30 01:12:26.226276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.226165    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-dshm\") pod \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") "
Oct 30 01:12:26.226276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.226202    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-model-cache\") pod \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") "
Oct 30 01:12:26.226489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.226305    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tls-certs\") pod \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") "
Oct 30 01:12:26.226489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.226335    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tmp-dir" pod "c4587c86-fe98-40d7-8473-30f6fb0f79dd" (UID: "c4587c86-fe98-40d7-8473-30f6fb0f79dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:26.226489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.226423    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-home" pod "c4587c86-fe98-40d7-8473-30f6fb0f79dd" (UID: "c4587c86-fe98-40d7-8473-30f6fb0f79dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:26.226655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.226521    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4587c86-fe98-40d7-8473-30f6fb0f79dd-kube-api-access-2tdft\" (UniqueName: \"kubernetes.io/projected/c4587c86-fe98-40d7-8473-30f6fb0f79dd-kube-api-access-2tdft\") pod \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\" (UID: \"c4587c86-fe98-40d7-8473-30f6fb0f79dd\") "
Oct 30 01:12:26.226655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.226575    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-model-cache" pod "c4587c86-fe98-40d7-8473-30f6fb0f79dd" (UID: "c4587c86-fe98-40d7-8473-30f6fb0f79dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:26.227395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.227208    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:26.227395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.227306    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:26.227395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.227324    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:26.228685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.228662    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-dshm" pod "c4587c86-fe98-40d7-8473-30f6fb0f79dd" (UID: "c4587c86-fe98-40d7-8473-30f6fb0f79dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:26.228958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.228944    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tls-certs" pod "c4587c86-fe98-40d7-8473-30f6fb0f79dd" (UID: "c4587c86-fe98-40d7-8473-30f6fb0f79dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:26.229331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.229315    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4587c86-fe98-40d7-8473-30f6fb0f79dd-kube-api-access-2tdft" pod "c4587c86-fe98-40d7-8473-30f6fb0f79dd" (UID: "c4587c86-fe98-40d7-8473-30f6fb0f79dd"). InnerVolumeSpecName "kube-api-access-2tdft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:26.328364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.328280    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4587c86-fe98-40d7-8473-30f6fb0f79dd-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:26.328364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.328311    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4587c86-fe98-40d7-8473-30f6fb0f79dd-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:26.328364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.328322    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tdft\" (UniqueName: \"kubernetes.io/projected/c4587c86-fe98-40d7-8473-30f6fb0f79dd-kube-api-access-2tdft\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:26.332763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.332653    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"
Oct 30 01:12:26.332763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.332686    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c" event={"ID":"d7f15448-71f1-4811-bd35-2c224f55d7eb","Type":"ContainerDied","Data":"710ebb87d9666ce63d164c0e444b7de76e877c382b401a0ae618ab26cef206e7"}
Oct 30 01:12:26.332763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.332738    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710ebb87d9666ce63d164c0e444b7de76e877c382b401a0ae618ab26cef206e7"
Oct 30 01:12:26.334546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.334521    4576 generic.go:356] "Generic (PLEG): container finished" podID="c4587c86-fe98-40d7-8473-30f6fb0f79dd" containerID="25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b" exitCode=0
Oct 30 01:12:26.334677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.334591    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"
Oct 30 01:12:26.334677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.334628    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr" event={"ID":"c4587c86-fe98-40d7-8473-30f6fb0f79dd","Type":"ContainerDied","Data":"25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b"}
Oct 30 01:12:26.334677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.334655    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr" event={"ID":"c4587c86-fe98-40d7-8473-30f6fb0f79dd","Type":"ContainerDied","Data":"3ad4dd065362df4aeac8e69e007e4c915f9005b25695dcbd29bdf7fc8a8254ee"}
Oct 30 01:12:26.334827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.334679    4576 scope.go:122] "RemoveContainer" containerID="25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b"
Oct 30 01:12:26.346410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.346389    4576 scope.go:122] "RemoveContainer" containerID="25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b"
Oct 30 01:12:26.346778 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:12:26.346756    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b\": container with ID starting with 25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b not found: ID does not exist" containerID="25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b"
Oct 30 01:12:26.346839 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.346788    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b"} err="failed to get container status \"25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b\": rpc error: code = NotFound desc = could not find container \"25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b\": container with ID starting with 25a0f7f2efaea5d2cabc649205dcda68c78e4172c75806d63a957ef81166bd7b not found: ID does not exist"
Oct 30 01:12:26.410734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.410692    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"]
Oct 30 01:12:26.416472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.416447    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b2bf8a/test-model-b2bf8a-kserve-75d7b44465-d2qhr"]
Oct 30 01:12:26.963382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:26.963350    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:28.110866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:28.110827    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4587c86-fe98-40d7-8473-30f6fb0f79dd" path="/var/lib/kubelet/pods/c4587c86-fe98-40d7-8473-30f6fb0f79dd/volumes"
Oct 30 01:12:28.824967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:28.824933    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:30.315144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:30.315102    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2" podUID="5c232f16-6971-4881-89a0-63a798665351" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:12:30.537513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:30.537471    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:31.630698 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:31.630666    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:31.746598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:31.746564    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:31.953692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:31.953653    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:32.406270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.406179    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"]
Oct 30 01:12:32.406866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.406849    4576 cpu_manager.go:414] "containerMap: removing container" podUID="c4587c86-fe98-40d7-8473-30f6fb0f79dd" containerName="main"
Oct 30 01:12:32.406915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.406868    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4587c86-fe98-40d7-8473-30f6fb0f79dd" containerName="main"
Oct 30 01:12:32.406915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.406900    4576 cpu_manager.go:414] "containerMap: removing container" podUID="d7f15448-71f1-4811-bd35-2c224f55d7eb" containerName="revoke-keys"
Oct 30 01:12:32.406915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.406906    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f15448-71f1-4811-bd35-2c224f55d7eb" containerName="revoke-keys"
Oct 30 01:12:32.407033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.407021    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4587c86-fe98-40d7-8473-30f6fb0f79dd" containerName="main"
Oct 30 01:12:32.407069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.407037    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f15448-71f1-4811-bd35-2c224f55d7eb" containerName="revoke-keys"
Oct 30 01:12:32.411972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.411955    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.428582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.428552    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"]
Oct 30 01:12:32.436923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.436892    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:32.437102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.436940    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:12:32.522361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.522331    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.522549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.522539    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.522624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.522602    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcd2\" (UniqueName: \"kubernetes.io/projected/e25eaa9b-c082-482d-af2f-d353745e9d82-kube-api-access-lpcd2\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.522954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.522927    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:32.624321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.624290    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.624527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.624357    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcd2\" (UniqueName: \"kubernetes.io/projected/e25eaa9b-c082-482d-af2f-d353745e9d82-kube-api-access-lpcd2\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.624527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.624487    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.625161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.625085    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.626843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.626819    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.645520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.645494    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcd2\" (UniqueName: \"kubernetes.io/projected/e25eaa9b-c082-482d-af2f-d353745e9d82-kube-api-access-lpcd2\") pod \"maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:32.727826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:32.727797    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:33.046332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:33.046287    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"]
Oct 30 01:12:33.047501 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:12:33.047469    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25eaa9b_c082_482d_af2f_d353745e9d82.slice/crio-f639febe4c90eacfdb386a16b5a0f41b8865c9d75cc7d9157e75990469960217 WatchSource:0}: Error finding container f639febe4c90eacfdb386a16b5a0f41b8865c9d75cc7d9157e75990469960217: Status 404 returned error can't find the container with id f639febe4c90eacfdb386a16b5a0f41b8865c9d75cc7d9157e75990469960217
Oct 30 01:12:33.400339 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:33.400263    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k" event={"ID":"e25eaa9b-c082-482d-af2f-d353745e9d82","Type":"ContainerStarted","Data":"f639febe4c90eacfdb386a16b5a0f41b8865c9d75cc7d9157e75990469960217"}
Oct 30 01:12:34.358078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:34.357989    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:34.420105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:34.420068    4576 generic.go:356] "Generic (PLEG): container finished" podID="e25eaa9b-c082-482d-af2f-d353745e9d82" containerID="16886c5c35ab72cea42ee2a0265b6d9defc68f48e0ebbbf9d6c692143888a7c7" exitCode=0
Oct 30 01:12:34.420313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:34.420195    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k" event={"ID":"e25eaa9b-c082-482d-af2f-d353745e9d82","Type":"ContainerDied","Data":"16886c5c35ab72cea42ee2a0265b6d9defc68f48e0ebbbf9d6c692143888a7c7"}
Oct 30 01:12:35.430928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.430898    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:35.606926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.606898    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:35.669938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.669904    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25eaa9b-c082-482d-af2f-d353745e9d82-kube-api-access-lpcd2\" (UniqueName: \"kubernetes.io/projected/e25eaa9b-c082-482d-af2f-d353745e9d82-kube-api-access-lpcd2\") pod \"e25eaa9b-c082-482d-af2f-d353745e9d82\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") "
Oct 30 01:12:35.670134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.669970    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-openshift-ca\") pod \"e25eaa9b-c082-482d-af2f-d353745e9d82\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") "
Oct 30 01:12:35.670134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.670010    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-xks-ca\") pod \"e25eaa9b-c082-482d-af2f-d353745e9d82\" (UID: \"e25eaa9b-c082-482d-af2f-d353745e9d82\") "
Oct 30 01:12:35.671048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.671016    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-openshift-ca" pod "e25eaa9b-c082-482d-af2f-d353745e9d82" (UID: "e25eaa9b-c082-482d-af2f-d353745e9d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:35.673724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.673697    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-xks-ca" pod "e25eaa9b-c082-482d-af2f-d353745e9d82" (UID: "e25eaa9b-c082-482d-af2f-d353745e9d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:35.674103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.674083    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25eaa9b-c082-482d-af2f-d353745e9d82-kube-api-access-lpcd2" pod "e25eaa9b-c082-482d-af2f-d353745e9d82" (UID: "e25eaa9b-c082-482d-af2f-d353745e9d82"). InnerVolumeSpecName "kube-api-access-lpcd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:35.772985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.772951    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpcd2\" (UniqueName: \"kubernetes.io/projected/e25eaa9b-c082-482d-af2f-d353745e9d82-kube-api-access-lpcd2\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:35.772985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.772984    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:35.773212 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:35.772999    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25eaa9b-c082-482d-af2f-d353745e9d82-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:36.093669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:36.093574    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:36.437775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:36.437746    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"
Oct 30 01:12:36.437775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:36.437756    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k" event={"ID":"e25eaa9b-c082-482d-af2f-d353745e9d82","Type":"ContainerDied","Data":"f639febe4c90eacfdb386a16b5a0f41b8865c9d75cc7d9157e75990469960217"}
Oct 30 01:12:36.438220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:36.437813    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f639febe4c90eacfdb386a16b5a0f41b8865c9d75cc7d9157e75990469960217"
Oct 30 01:12:36.730569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:36.730461    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44" podUID="2678af68-73b3-4ec2-a4e4-03e42c73b45b" containerName="main" containerID="cri-o://180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903" gracePeriod=45
Oct 30 01:12:36.983415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:36.983311    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:37.080975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.080947    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:12:37.192412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.192381    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-home\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-home\") pod \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") "
Oct 30 01:12:37.192551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.192451    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tls-certs\") pod \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") "
Oct 30 01:12:37.192551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.192490    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-dshm\") pod \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") "
Oct 30 01:12:37.192664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.192601    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2678af68-73b3-4ec2-a4e4-03e42c73b45b-kube-api-access-ndvnp\" (UniqueName: \"kubernetes.io/projected/2678af68-73b3-4ec2-a4e4-03e42c73b45b-kube-api-access-ndvnp\") pod \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") "
Oct 30 01:12:37.192664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.192635    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tmp-dir\") pod \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") "
Oct 30 01:12:37.192788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.192772    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-model-cache\") pod \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\" (UID: \"2678af68-73b3-4ec2-a4e4-03e42c73b45b\") "
Oct 30 01:12:37.194140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.194109    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-home" pod "2678af68-73b3-4ec2-a4e4-03e42c73b45b" (UID: "2678af68-73b3-4ec2-a4e4-03e42c73b45b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:37.214075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.199337    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-model-cache" pod "2678af68-73b3-4ec2-a4e4-03e42c73b45b" (UID: "2678af68-73b3-4ec2-a4e4-03e42c73b45b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:37.214075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.199572    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tmp-dir" pod "2678af68-73b3-4ec2-a4e4-03e42c73b45b" (UID: "2678af68-73b3-4ec2-a4e4-03e42c73b45b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:37.216692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.216655    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tls-certs" pod "2678af68-73b3-4ec2-a4e4-03e42c73b45b" (UID: "2678af68-73b3-4ec2-a4e4-03e42c73b45b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:37.216811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.216696    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-dshm" pod "2678af68-73b3-4ec2-a4e4-03e42c73b45b" (UID: "2678af68-73b3-4ec2-a4e4-03e42c73b45b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:37.219398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.219369    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2678af68-73b3-4ec2-a4e4-03e42c73b45b-kube-api-access-ndvnp" pod "2678af68-73b3-4ec2-a4e4-03e42c73b45b" (UID: "2678af68-73b3-4ec2-a4e4-03e42c73b45b"). InnerVolumeSpecName "kube-api-access-ndvnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:37.295707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.295670    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.295707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.295709    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndvnp\" (UniqueName: \"kubernetes.io/projected/2678af68-73b3-4ec2-a4e4-03e42c73b45b-kube-api-access-ndvnp\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.296003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.295725    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.296003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.295740    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.296003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.295752    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2678af68-73b3-4ec2-a4e4-03e42c73b45b-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.296003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.295764    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2678af68-73b3-4ec2-a4e4-03e42c73b45b-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.310438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.310407    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:12:37.397315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.397278    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c232f16-6971-4881-89a0-63a798665351-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c232f16-6971-4881-89a0-63a798665351-plugins-config-volume\") pod \"5c232f16-6971-4881-89a0-63a798665351\" (UID: \"5c232f16-6971-4881-89a0-63a798665351\") "
Oct 30 01:12:37.397524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.397350    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c232f16-6971-4881-89a0-63a798665351-kube-api-access-t2nnl\" (UniqueName: \"kubernetes.io/projected/5c232f16-6971-4881-89a0-63a798665351-kube-api-access-t2nnl\") pod \"5c232f16-6971-4881-89a0-63a798665351\" (UID: \"5c232f16-6971-4881-89a0-63a798665351\") "
Oct 30 01:12:37.397871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.397820    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c232f16-6971-4881-89a0-63a798665351-plugins-config-volume" pod "5c232f16-6971-4881-89a0-63a798665351" (UID: "5c232f16-6971-4881-89a0-63a798665351"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:37.399061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.398309    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c232f16-6971-4881-89a0-63a798665351-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.400430 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.400389    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c232f16-6971-4881-89a0-63a798665351-kube-api-access-t2nnl" pod "5c232f16-6971-4881-89a0-63a798665351" (UID: "5c232f16-6971-4881-89a0-63a798665351"). InnerVolumeSpecName "kube-api-access-t2nnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:37.447285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.447255    4576 generic.go:356] "Generic (PLEG): container finished" podID="5c232f16-6971-4881-89a0-63a798665351" containerID="b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4" exitCode=0
Oct 30 01:12:37.447722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.447313    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"
Oct 30 01:12:37.447722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.447328    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2" event={"ID":"5c232f16-6971-4881-89a0-63a798665351","Type":"ContainerDied","Data":"b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4"}
Oct 30 01:12:37.447722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.447387    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2" event={"ID":"5c232f16-6971-4881-89a0-63a798665351","Type":"ContainerDied","Data":"6b4d580340b0229aa0af501e729916830cb4801d5c445072c9a63b35a429493b"}
Oct 30 01:12:37.447722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.447413    4576 scope.go:122] "RemoveContainer" containerID="b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4"
Oct 30 01:12:37.449268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.449212    4576 generic.go:356] "Generic (PLEG): container finished" podID="2678af68-73b3-4ec2-a4e4-03e42c73b45b" containerID="180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903" exitCode=0
Oct 30 01:12:37.449382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.449266    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44" event={"ID":"2678af68-73b3-4ec2-a4e4-03e42c73b45b","Type":"ContainerDied","Data":"180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903"}
Oct 30 01:12:37.449382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.449280    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"
Oct 30 01:12:37.449382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.449311    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44" event={"ID":"2678af68-73b3-4ec2-a4e4-03e42c73b45b","Type":"ContainerDied","Data":"7f73f7c87c130fac82d5990d4a79bdcccbad835b83fe966b06261e159ebf6382"}
Oct 30 01:12:37.460861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.460836    4576 scope.go:122] "RemoveContainer" containerID="b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4"
Oct 30 01:12:37.461189 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:12:37.461154    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4\": container with ID starting with b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4 not found: ID does not exist" containerID="b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4"
Oct 30 01:12:37.461294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.461192    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4"} err="failed to get container status \"b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4\": rpc error: code = NotFound desc = could not find container \"b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4\": container with ID starting with b07b01b438ceb064a64a2c8dc163f2a6fd12bb125f1ef4f20df42ccf989c19a4 not found: ID does not exist"
Oct 30 01:12:37.461294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.461213    4576 scope.go:122] "RemoveContainer" containerID="180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903"
Oct 30 01:12:37.471576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.471555    4576 scope.go:122] "RemoveContainer" containerID="180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903"
Oct 30 01:12:37.471860 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:12:37.471840    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903\": container with ID starting with 180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903 not found: ID does not exist" containerID="180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903"
Oct 30 01:12:37.471933 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.471872    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903"} err="failed to get container status \"180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903\": rpc error: code = NotFound desc = could not find container \"180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903\": container with ID starting with 180dd82565e271977bd4418bcadd123dd2468a3337aea5ff79430ef4cca48903 not found: ID does not exist"
Oct 30 01:12:37.501073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.500986    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2nnl\" (UniqueName: \"kubernetes.io/projected/5c232f16-6971-4881-89a0-63a798665351-kube-api-access-t2nnl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.507896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.507863    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"]
Oct 30 01:12:37.511420 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.511398    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44"]
Oct 30 01:12:37.546928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.546893    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"]
Oct 30 01:12:37.550501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.550478    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1eae0b-5b589f764c-qltf2"]
Oct 30 01:12:37.873011 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:37.872932    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:38.064508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:38.064465    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0680fa/test-model-0680fa-kserve-c67cdc59c-6gb44" podUID="2678af68-73b3-4ec2-a4e4-03e42c73b45b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Oct 30 01:12:38.111243 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:38.111193    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2678af68-73b3-4ec2-a4e4-03e42c73b45b" path="/var/lib/kubelet/pods/2678af68-73b3-4ec2-a4e4-03e42c73b45b/volumes"
Oct 30 01:12:38.111672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:38.111655    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c232f16-6971-4881-89a0-63a798665351" path="/var/lib/kubelet/pods/5c232f16-6971-4881-89a0-63a798665351/volumes"
Oct 30 01:12:39.552260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.551213    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:39.583541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.583510    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"]
Oct 30 01:12:39.698257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.690583    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:39.763313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.763275    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"]
Oct 30 01:12:39.765400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.765370    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e25eaa9b-c082-482d-af2f-d353745e9d82" containerName="revoke-keys"
Oct 30 01:12:39.765400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.765398    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25eaa9b-c082-482d-af2f-d353745e9d82" containerName="revoke-keys"
Oct 30 01:12:39.765602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.765485    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5c232f16-6971-4881-89a0-63a798665351" containerName="payload-pre-processing"
Oct 30 01:12:39.765602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.765494    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c232f16-6971-4881-89a0-63a798665351" containerName="payload-pre-processing"
Oct 30 01:12:39.765602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.765522    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2678af68-73b3-4ec2-a4e4-03e42c73b45b" containerName="main"
Oct 30 01:12:39.765602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.765531    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2678af68-73b3-4ec2-a4e4-03e42c73b45b" containerName="main"
Oct 30 01:12:39.766008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.765991    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c232f16-6971-4881-89a0-63a798665351" containerName="payload-pre-processing"
Oct 30 01:12:39.766098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.766012    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2678af68-73b3-4ec2-a4e4-03e42c73b45b" containerName="main"
Oct 30 01:12:39.766098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.766045    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25eaa9b-c082-482d-af2f-d353745e9d82" containerName="revoke-keys"
Oct 30 01:12:39.779255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.774748    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:39.811812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.811727    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"]
Oct 30 01:12:39.813654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.813590    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:39.813805 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.813713    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:12:39.878269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.878200    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:39.878446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.878293    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fh6f\" (UniqueName: \"kubernetes.io/projected/c11a240f-abdd-473c-9482-7893fcb3857e-kube-api-access-5fh6f\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:39.878446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.878338    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:39.979637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.979593    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:39.979844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.979712    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fh6f\" (UniqueName: \"kubernetes.io/projected/c11a240f-abdd-473c-9482-7893fcb3857e-kube-api-access-5fh6f\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:39.979844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.979775    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:39.983268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.981592    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:39.983568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:39.983540    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:40.008313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:40.008273    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fh6f\" (UniqueName: \"kubernetes.io/projected/c11a240f-abdd-473c-9482-7893fcb3857e-kube-api-access-5fh6f\") pod \"maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:40.137491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:40.137412    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:12:40.139466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:40.139445    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:40.455564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:40.455538    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"]
Oct 30 01:12:40.455700 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:12:40.455544    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11a240f_abdd_473c_9482_7893fcb3857e.slice/crio-550fba9f414c38ddf1998fbdf2ee3f1264368a3dc8e37d72a9ba8578cfc6cfb8 WatchSource:0}: Error finding container 550fba9f414c38ddf1998fbdf2ee3f1264368a3dc8e37d72a9ba8578cfc6cfb8: Status 404 returned error can't find the container with id 550fba9f414c38ddf1998fbdf2ee3f1264368a3dc8e37d72a9ba8578cfc6cfb8
Oct 30 01:12:40.481412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:40.481382    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6" event={"ID":"c11a240f-abdd-473c-9482-7893fcb3857e","Type":"ContainerStarted","Data":"550fba9f414c38ddf1998fbdf2ee3f1264368a3dc8e37d72a9ba8578cfc6cfb8"}
Oct 30 01:12:40.773752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:40.773726    4576 scope.go:122] "RemoveContainer" containerID="f2ac475a0c9acc0bda59d1178bc3e28da75fee068d46ee9cef653a0d977b26d1"
Oct 30 01:12:40.784085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:40.784063    4576 scope.go:122] "RemoveContainer" containerID="4870b5e048d9408b0bf04516b9a21ac188abe78479567905948e33335a8e8b3b"
Oct 30 01:12:41.490986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:41.490950    4576 generic.go:356] "Generic (PLEG): container finished" podID="c11a240f-abdd-473c-9482-7893fcb3857e" containerID="18c5969c5c684ec1ccb6459fc04228fd94d4201d780cc57b1b444532a89521a8" exitCode=0
Oct 30 01:12:41.491145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:41.491034    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6" event={"ID":"c11a240f-abdd-473c-9482-7893fcb3857e","Type":"ContainerDied","Data":"18c5969c5c684ec1ccb6459fc04228fd94d4201d780cc57b1b444532a89521a8"}
Oct 30 01:12:41.773586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:41.773545    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:42.663850 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.663824    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:42.721221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.721183    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11a240f-abdd-473c-9482-7893fcb3857e-kube-api-access-5fh6f\" (UniqueName: \"kubernetes.io/projected/c11a240f-abdd-473c-9482-7893fcb3857e-kube-api-access-5fh6f\") pod \"c11a240f-abdd-473c-9482-7893fcb3857e\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") "
Oct 30 01:12:42.721506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.721482    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-xks-ca\") pod \"c11a240f-abdd-473c-9482-7893fcb3857e\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") "
Oct 30 01:12:42.721744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.721724    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-openshift-ca\") pod \"c11a240f-abdd-473c-9482-7893fcb3857e\" (UID: \"c11a240f-abdd-473c-9482-7893fcb3857e\") "
Oct 30 01:12:42.722361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.722324    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-openshift-ca" pod "c11a240f-abdd-473c-9482-7893fcb3857e" (UID: "c11a240f-abdd-473c-9482-7893fcb3857e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:42.723757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.723728    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11a240f-abdd-473c-9482-7893fcb3857e-kube-api-access-5fh6f" pod "c11a240f-abdd-473c-9482-7893fcb3857e" (UID: "c11a240f-abdd-473c-9482-7893fcb3857e"). InnerVolumeSpecName "kube-api-access-5fh6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:42.724108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.724082    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-xks-ca" pod "c11a240f-abdd-473c-9482-7893fcb3857e" (UID: "c11a240f-abdd-473c-9482-7893fcb3857e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:42.823761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.823665    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:42.823761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.823700    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fh6f\" (UniqueName: \"kubernetes.io/projected/c11a240f-abdd-473c-9482-7893fcb3857e-kube-api-access-5fh6f\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:42.823761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.823714    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11a240f-abdd-473c-9482-7893fcb3857e-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:42.889075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:42.889042    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"]
Oct 30 01:12:43.016982 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:43.016948    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:43.138290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:43.138158    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:43.328917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:43.328879    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:43.538254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:43.529296    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6" event={"ID":"c11a240f-abdd-473c-9482-7893fcb3857e","Type":"ContainerDied","Data":"550fba9f414c38ddf1998fbdf2ee3f1264368a3dc8e37d72a9ba8578cfc6cfb8"}
Oct 30 01:12:43.538254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:43.529394    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="550fba9f414c38ddf1998fbdf2ee3f1264368a3dc8e37d72a9ba8578cfc6cfb8"
Oct 30 01:12:43.538254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:43.529487    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"
Oct 30 01:12:44.976059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:44.976022    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:46.479957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:46.479922    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:47.890119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:47.890081    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl" podUID="556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" containerName="payload-pre-processing" containerID="cri-o://27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31" gracePeriod=25
Oct 30 01:12:48.322052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:48.322003    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:48.390542 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:12:48.390498    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:12:49.556172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:49.556136    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl" podUID="556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:12:49.823056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:49.822971    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:50.368017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:50.367985    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:50.509961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:50.509922    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:50.558656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:50.558620    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"]
Oct 30 01:12:50.569330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:50.569298    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1eae0b-q2xmj"]
Oct 30 01:12:50.627122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:50.627039    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:51.040356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:51.040322    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:52.109792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:52.109760    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a6012a-c71a-4df8-b7df-3491b6527a95" path="/var/lib/kubelet/pods/55a6012a-c71a-4df8-b7df-3491b6527a95/volumes"
Oct 30 01:12:52.221919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:52.221884    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:52.481027 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:52.481000    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:54.068692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:54.068653    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:54.606089 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:54.606043    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl" podUID="32f7439a-4727-4f66-99e6-716ca8003fc1" containerName="main" containerID="cri-o://a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f" gracePeriod=45
Oct 30 01:12:54.889316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:54.889292    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:12:55.062914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.062875    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-home\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-home\") pod \"32f7439a-4727-4f66-99e6-716ca8003fc1\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") "
Oct 30 01:12:55.063174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.063147    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-home" pod "32f7439a-4727-4f66-99e6-716ca8003fc1" (UID: "32f7439a-4727-4f66-99e6-716ca8003fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:55.063325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.063135    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f7439a-4727-4f66-99e6-716ca8003fc1-kube-api-access-pvql8\" (UniqueName: \"kubernetes.io/projected/32f7439a-4727-4f66-99e6-716ca8003fc1-kube-api-access-pvql8\") pod \"32f7439a-4727-4f66-99e6-716ca8003fc1\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") "
Oct 30 01:12:55.063459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.063390    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f7439a-4727-4f66-99e6-716ca8003fc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7439a-4727-4f66-99e6-716ca8003fc1-tls-certs\") pod \"32f7439a-4727-4f66-99e6-716ca8003fc1\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") "
Oct 30 01:12:55.063528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.063459    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-model-cache\") pod \"32f7439a-4727-4f66-99e6-716ca8003fc1\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") "
Oct 30 01:12:55.063528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.063491    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-tmp-dir\") pod \"32f7439a-4727-4f66-99e6-716ca8003fc1\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") "
Oct 30 01:12:55.063643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.063603    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-dshm\") pod \"32f7439a-4727-4f66-99e6-716ca8003fc1\" (UID: \"32f7439a-4727-4f66-99e6-716ca8003fc1\") "
Oct 30 01:12:55.064056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.064032    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-tmp-dir" pod "32f7439a-4727-4f66-99e6-716ca8003fc1" (UID: "32f7439a-4727-4f66-99e6-716ca8003fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:55.064397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.064378    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.064540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.064528    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.064868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.064839    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-model-cache" pod "32f7439a-4727-4f66-99e6-716ca8003fc1" (UID: "32f7439a-4727-4f66-99e6-716ca8003fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:55.066330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.066306    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f7439a-4727-4f66-99e6-716ca8003fc1-tls-certs" pod "32f7439a-4727-4f66-99e6-716ca8003fc1" (UID: "32f7439a-4727-4f66-99e6-716ca8003fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:55.066504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.066473    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f7439a-4727-4f66-99e6-716ca8003fc1-kube-api-access-pvql8" pod "32f7439a-4727-4f66-99e6-716ca8003fc1" (UID: "32f7439a-4727-4f66-99e6-716ca8003fc1"). InnerVolumeSpecName "kube-api-access-pvql8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:55.066586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.066561    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-dshm" pod "32f7439a-4727-4f66-99e6-716ca8003fc1" (UID: "32f7439a-4727-4f66-99e6-716ca8003fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:55.166282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.166139    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.166282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.166202    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7439a-4727-4f66-99e6-716ca8003fc1-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.166282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.166214    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvql8\" (UniqueName: \"kubernetes.io/projected/32f7439a-4727-4f66-99e6-716ca8003fc1-kube-api-access-pvql8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.166282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.166224    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7439a-4727-4f66-99e6-716ca8003fc1-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.625634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.625598    4576 generic.go:356] "Generic (PLEG): container finished" podID="32f7439a-4727-4f66-99e6-716ca8003fc1" containerID="a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f" exitCode=0
Oct 30 01:12:55.625823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.625664    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"
Oct 30 01:12:55.625823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.625688    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl" event={"ID":"32f7439a-4727-4f66-99e6-716ca8003fc1","Type":"ContainerDied","Data":"a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f"}
Oct 30 01:12:55.625823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.625731    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl" event={"ID":"32f7439a-4727-4f66-99e6-716ca8003fc1","Type":"ContainerDied","Data":"2e50c7f096dad8dd8c164dfff022d0656bfa339ede3f835ba052a304cd4d4228"}
Oct 30 01:12:55.625823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.625753    4576 scope.go:122] "RemoveContainer" containerID="a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f"
Oct 30 01:12:55.636978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.636956    4576 scope.go:122] "RemoveContainer" containerID="a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f"
Oct 30 01:12:55.637313 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:12:55.637291    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f\": container with ID starting with a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f not found: ID does not exist" containerID="a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f"
Oct 30 01:12:55.637425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.637322    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f"} err="failed to get container status \"a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f\": rpc error: code = NotFound desc = could not find container \"a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f\": container with ID starting with a67ee2fdede60c91f4c11f4119423ae085a0592f529975bb1fa67d05c5d2f12f not found: ID does not exist"
Oct 30 01:12:55.687083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.687040    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"]
Oct 30 01:12:55.691197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.691171    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d54608/test-model-d54608-kserve-684448976f-d88sl"]
Oct 30 01:12:55.866023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:55.865984    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:56.110376 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:56.110340    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f7439a-4727-4f66-99e6-716ca8003fc1" path="/var/lib/kubelet/pods/32f7439a-4727-4f66-99e6-716ca8003fc1/volumes"
Oct 30 01:12:57.250386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:57.250353    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:57.368935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:57.368899    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:57.484864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:57.484829    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:57.529662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:57.529571    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"]
Oct 30 01:12:57.529876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:57.529844    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy" containerID="cri-o://1e8537b88381ab1281353fc9c007c88710baa34e7bf34e342afc56fa324d619d" gracePeriod=30
Oct 30 01:12:57.609343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:57.609298    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:58.361089 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:58.361057    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:58.834766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:58.834733    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:58.960258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:58.952802    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:59.062802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:59.062724    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:12:59.555403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:12:59.555362    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl" podUID="556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:13:00.005200 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.005161    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:13:00.787822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.787788    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"]
Oct 30 01:13:00.788437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.788422    4576 cpu_manager.go:414] "containerMap: removing container" podUID="32f7439a-4727-4f66-99e6-716ca8003fc1" containerName="main"
Oct 30 01:13:00.788514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.788438    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f7439a-4727-4f66-99e6-716ca8003fc1" containerName="main"
Oct 30 01:13:00.788514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.788448    4576 cpu_manager.go:414] "containerMap: removing container" podUID="c11a240f-abdd-473c-9482-7893fcb3857e" containerName="revoke-keys"
Oct 30 01:13:00.788514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.788453    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11a240f-abdd-473c-9482-7893fcb3857e" containerName="revoke-keys"
Oct 30 01:13:00.788621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.788595    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f7439a-4727-4f66-99e6-716ca8003fc1" containerName="main"
Oct 30 01:13:00.788621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.788605    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11a240f-abdd-473c-9482-7893fcb3857e" containerName="revoke-keys"
Oct 30 01:13:00.793666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.793645    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:00.809027 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.809006    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"]
Oct 30 01:13:00.809691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.809672    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:00.817766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.817739    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:13:00.870394 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.870362    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:00.877964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.877938    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:00.878153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.878011    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:00.878220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.878192    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np6cn\" (UniqueName: \"kubernetes.io/projected/26cfa32f-a497-44dc-8591-22ae8c972694-kube-api-access-np6cn\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:00.979984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.979953    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np6cn\" (UniqueName: \"kubernetes.io/projected/26cfa32f-a497-44dc-8591-22ae8c972694-kube-api-access-np6cn\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:00.980162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.980057    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:00.980162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.980110    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:00.980826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.980797    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:00.982711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:00.982690    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:01.004254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:01.001522    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np6cn\" (UniqueName: \"kubernetes.io/projected/26cfa32f-a497-44dc-8591-22ae8c972694-kube-api-access-np6cn\") pod \"maas-api-revoke-sub-2f37044b-jbf8m\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:01.111796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:01.111474    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:01.637512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:01.637425    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"]
Oct 30 01:13:01.641851 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:01.641723    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cfa32f_a497_44dc_8591_22ae8c972694.slice/crio-cd4693f6b68c2577810bfaf29dc19123446700cace95470ff287144bbd3813ee WatchSource:0}: Error finding container cd4693f6b68c2577810bfaf29dc19123446700cace95470ff287144bbd3813ee: Status 404 returned error can't find the container with id cd4693f6b68c2577810bfaf29dc19123446700cace95470ff287144bbd3813ee
Oct 30 01:13:01.677204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:01.677168    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m" event={"ID":"26cfa32f-a497-44dc-8591-22ae8c972694","Type":"ContainerStarted","Data":"cd4693f6b68c2577810bfaf29dc19123446700cace95470ff287144bbd3813ee"}
Oct 30 01:13:02.690121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.688432    4576 generic.go:356] "Generic (PLEG): container finished" podID="26cfa32f-a497-44dc-8591-22ae8c972694" containerID="06444b2b780f0bca31f1543a3df8d28bb414435afbfd4966d4d28a5fe727396e" exitCode=0
Oct 30 01:13:02.690121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.688510    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m" event={"ID":"26cfa32f-a497-44dc-8591-22ae8c972694","Type":"ContainerDied","Data":"06444b2b780f0bca31f1543a3df8d28bb414435afbfd4966d4d28a5fe727396e"}
Oct 30 01:13:02.690925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.690898    4576 generic.go:356] "Generic (PLEG): container finished" podID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerID="1e8537b88381ab1281353fc9c007c88710baa34e7bf34e342afc56fa324d619d" exitCode=0
Oct 30 01:13:02.690999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.690951    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" event={"ID":"228a0ad5-07f7-43ba-bc33-a2a61a97996f","Type":"ContainerDied","Data":"1e8537b88381ab1281353fc9c007c88710baa34e7bf34e342afc56fa324d619d"}
Oct 30 01:13:02.691057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.691001    4576 scope.go:122] "RemoveContainer" containerID="430bedfb19cc8c2e3ff10259ff2071cc4c2755e5d280c7c1a9626d95b8a0645b"
Oct 30 01:13:02.839091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.839067    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:13:02.947062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947027    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-token\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-token\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.947273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947109    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-certs\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.947273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947143    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-envoy\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.947273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947213    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-podinfo\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.947455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947398    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-kube-api-access-m8hnp\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.947513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947461    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-data\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.947513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947459    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-certs" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:02.947513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947499    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-socket\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.947699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947622    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istiod-ca-cert\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.947699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947656    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-credential-socket\") pod \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\" (UID: \"228a0ad5-07f7-43ba-bc33-a2a61a97996f\") "
Oct 30 01:13:02.948303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.947897    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-socket" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:02.948303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.948075    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-data" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:02.948491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.948323    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-credential-socket" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:02.948491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.948468    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:02.948491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.948486    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:02.948675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.948500    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:02.948675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.948515    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:02.948784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.948697    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istiod-ca-cert" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:02.949851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.949822    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-envoy" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:02.950192 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.950173    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-token" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:02.950782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.950765    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-kube-api-access-m8hnp" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "kube-api-access-m8hnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:02.950954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:02.950939    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-podinfo" pod "228a0ad5-07f7-43ba-bc33-a2a61a97996f" (UID: "228a0ad5-07f7-43ba-bc33-a2a61a97996f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:03.049752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.049720    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:03.049752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.049749    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:03.049752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.049757    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:03.050004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.049765    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228a0ad5-07f7-43ba-bc33-a2a61a97996f-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:03.050004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.049774    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hnp\" (UniqueName: \"kubernetes.io/projected/228a0ad5-07f7-43ba-bc33-a2a61a97996f-kube-api-access-m8hnp\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:03.106363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.106272    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:03.453198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.453162    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"]
Oct 30 01:13:03.453437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.453411    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy" containerID="cri-o://b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25" gracePeriod=30
Oct 30 01:13:03.508745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.508712    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:03.638428 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.638395    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:03.700979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.700946    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"
Oct 30 01:13:03.700979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.700960    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w" event={"ID":"228a0ad5-07f7-43ba-bc33-a2a61a97996f","Type":"ContainerDied","Data":"92a708876537a0ef3b7afbd54258164beaebbbb9b0b7d9ce8c970c88d3bad391"}
Oct 30 01:13:03.701759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.701029    4576 scope.go:122] "RemoveContainer" containerID="1e8537b88381ab1281353fc9c007c88710baa34e7bf34e342afc56fa324d619d"
Oct 30 01:13:03.794294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.793648    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"]
Oct 30 01:13:03.798939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.797833    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1eae0b-openshift-default-cf964958f-w862w"]
Oct 30 01:13:03.883569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.883538    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:03.956921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.956887    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"]
Oct 30 01:13:03.957479 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.957444    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy" containerID="cri-o://dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15" gracePeriod=30
Oct 30 01:13:03.962611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.962510    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-openshift-ca\") pod \"26cfa32f-a497-44dc-8591-22ae8c972694\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") "
Oct 30 01:13:03.962611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.962583    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cfa32f-a497-44dc-8591-22ae8c972694-kube-api-access-np6cn\" (UniqueName: \"kubernetes.io/projected/26cfa32f-a497-44dc-8591-22ae8c972694-kube-api-access-np6cn\") pod \"26cfa32f-a497-44dc-8591-22ae8c972694\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") "
Oct 30 01:13:03.962996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.962624    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-xks-ca\") pod \"26cfa32f-a497-44dc-8591-22ae8c972694\" (UID: \"26cfa32f-a497-44dc-8591-22ae8c972694\") "
Oct 30 01:13:03.963694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.963380    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-openshift-ca" pod "26cfa32f-a497-44dc-8591-22ae8c972694" (UID: "26cfa32f-a497-44dc-8591-22ae8c972694"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:03.966280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.966041    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-xks-ca" pod "26cfa32f-a497-44dc-8591-22ae8c972694" (UID: "26cfa32f-a497-44dc-8591-22ae8c972694"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:03.966280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:03.966175    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cfa32f-a497-44dc-8591-22ae8c972694-kube-api-access-np6cn" pod "26cfa32f-a497-44dc-8591-22ae8c972694" (UID: "26cfa32f-a497-44dc-8591-22ae8c972694"). InnerVolumeSpecName "kube-api-access-np6cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:04.065518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.065484    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:04.065518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.065511    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np6cn\" (UniqueName: \"kubernetes.io/projected/26cfa32f-a497-44dc-8591-22ae8c972694-kube-api-access-np6cn\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:04.065518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.065523    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26cfa32f-a497-44dc-8591-22ae8c972694-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:04.111777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.111733    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" path="/var/lib/kubelet/pods/228a0ad5-07f7-43ba-bc33-a2a61a97996f/volumes"
Oct 30 01:13:04.112569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.112549    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"]
Oct 30 01:13:04.112789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.112763    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy" containerID="cri-o://18ed205a8bffeeba972efab4c7763351d6f56ff28264ec4943e1a1cc424eda36" gracePeriod=30
Oct 30 01:13:04.711098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.711062    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"
Oct 30 01:13:04.711098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.711086    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m" event={"ID":"26cfa32f-a497-44dc-8591-22ae8c972694","Type":"ContainerDied","Data":"cd4693f6b68c2577810bfaf29dc19123446700cace95470ff287144bbd3813ee"}
Oct 30 01:13:04.711670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:04.711132    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd4693f6b68c2577810bfaf29dc19123446700cace95470ff287144bbd3813ee"
Oct 30 01:13:05.026285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.026185    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"]
Oct 30 01:13:05.027106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.027089    4576 cpu_manager.go:414] "containerMap: removing container" podUID="26cfa32f-a497-44dc-8591-22ae8c972694" containerName="revoke-keys"
Oct 30 01:13:05.027165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.027109    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cfa32f-a497-44dc-8591-22ae8c972694" containerName="revoke-keys"
Oct 30 01:13:05.027165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.027160    4576 cpu_manager.go:414] "containerMap: removing container" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy"
Oct 30 01:13:05.027551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.027167    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy"
Oct 30 01:13:05.027551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.027188    4576 cpu_manager.go:414] "containerMap: removing container" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy"
Oct 30 01:13:05.027551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.027196    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy"
Oct 30 01:13:05.027551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.027344    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy"
Oct 30 01:13:05.027551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.027362    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cfa32f-a497-44dc-8591-22ae8c972694" containerName="revoke-keys"
Oct 30 01:13:05.031706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.031686    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.051577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.051545    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:05.057874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.057835    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"]
Oct 30 01:13:05.058395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.058369    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:13:05.080487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.080455    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9f4\" (UniqueName: \"kubernetes.io/projected/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-kube-api-access-sk9f4\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.080737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.080615    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.080841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.080822    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.182526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.182486    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.182699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.182541    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9f4\" (UniqueName: \"kubernetes.io/projected/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-kube-api-access-sk9f4\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.182699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.182647    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.183362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.183342    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.185023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.185004    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.204967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.204933    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9f4\" (UniqueName: \"kubernetes.io/projected/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-kube-api-access-sk9f4\") pod \"maas-api-revoke-sub-8cd6e6d0-jhfzx\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.228409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.228369    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:05.339758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.339654    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:05.356098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.356067    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:05.696255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.694144    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"]
Oct 30 01:13:05.713210 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:05.713166    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaae8a5_6ca0_484f_a6c7_b230eb34a7ed.slice/crio-f8f772ba40d320f6323a1870863e2ca36ca65862ef5d479aab5bdf0e974e55bc WatchSource:0}: Error finding container f8f772ba40d320f6323a1870863e2ca36ca65862ef5d479aab5bdf0e974e55bc: Status 404 returned error can't find the container with id f8f772ba40d320f6323a1870863e2ca36ca65862ef5d479aab5bdf0e974e55bc
Oct 30 01:13:05.722983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:05.722951    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx" event={"ID":"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed","Type":"ContainerStarted","Data":"f8f772ba40d320f6323a1870863e2ca36ca65862ef5d479aab5bdf0e974e55bc"}
Oct 30 01:13:06.614088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:06.614055    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:06.734987 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:06.734955    4576 generic.go:356] "Generic (PLEG): container finished" podID="2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed" containerID="a43c8aec59226858afeba41956d631ee2ed47cf09e69e170beee7f8d81a2aefa" exitCode=0
Oct 30 01:13:06.735821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:06.735051    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx" event={"ID":"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed","Type":"ContainerDied","Data":"a43c8aec59226858afeba41956d631ee2ed47cf09e69e170beee7f8d81a2aefa"}
Oct 30 01:13:07.297216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:07.297180    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:07.989743 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:07.989712    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:08.046438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.045910    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-kube-api-access-sk9f4\" (UniqueName: \"kubernetes.io/projected/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-kube-api-access-sk9f4\") pod \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") "
Oct 30 01:13:08.046438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.045977    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-xks-ca\") pod \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") "
Oct 30 01:13:08.046438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.046226    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-openshift-ca\") pod \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\" (UID: \"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed\") "
Oct 30 01:13:08.046818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.046782    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-openshift-ca" pod "2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed" (UID: "2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:08.048983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.048958    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-kube-api-access-sk9f4" pod "2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed" (UID: "2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed"). InnerVolumeSpecName "kube-api-access-sk9f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:08.065111 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.065073    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-xks-ca" pod "2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed" (UID: "2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:08.148951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.148913    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9f4\" (UniqueName: \"kubernetes.io/projected/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-kube-api-access-sk9f4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.148951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.148953    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.149164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.148968    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.154833 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.154794    4576 patch_prober.go:29] interesting pod/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:13:08.154985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.154862    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:13:08.183459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.183434    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:13:08.250120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.250091    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-kube-api-access-6992x\" (UniqueName: \"kubernetes.io/projected/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-kube-api-access-6992x\") pod \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\" (UID: \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\") "
Oct 30 01:13:08.250322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.250147    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-plugins-config-volume\") pod \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\" (UID: \"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75\") "
Oct 30 01:13:08.250626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.250591    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-plugins-config-volume" pod "556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" (UID: "556fdf7e-9b7d-483b-b3b6-04fa78ca3b75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:08.250985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.250966    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.252578 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.252554    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-kube-api-access-6992x" pod "556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" (UID: "556fdf7e-9b7d-483b-b3b6-04fa78ca3b75"). InnerVolumeSpecName "kube-api-access-6992x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:08.352555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.352474    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6992x\" (UniqueName: \"kubernetes.io/projected/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75-kube-api-access-6992x\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.722673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.722649    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 01:13:08.750437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.750403    4576 generic.go:356] "Generic (PLEG): container finished" podID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerID="b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25" exitCode=0
Oct 30 01:13:08.750600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.750446    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"
Oct 30 01:13:08.750600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.750485    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" event={"ID":"dbc5c913-545b-43c0-ac8d-ab314d7ffb77","Type":"ContainerDied","Data":"b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25"}
Oct 30 01:13:08.750600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.750545    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x" event={"ID":"dbc5c913-545b-43c0-ac8d-ab314d7ffb77","Type":"ContainerDied","Data":"097144491ed6305cc0246c1596a57576fb20d0311ca74656aa7fd8ae4decb7fb"}
Oct 30 01:13:08.750600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.750570    4576 scope.go:122] "RemoveContainer" containerID="b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25"
Oct 30 01:13:08.752307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.752287    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"
Oct 30 01:13:08.752455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.752337    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx" event={"ID":"2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed","Type":"ContainerDied","Data":"f8f772ba40d320f6323a1870863e2ca36ca65862ef5d479aab5bdf0e974e55bc"}
Oct 30 01:13:08.752455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.752386    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f772ba40d320f6323a1870863e2ca36ca65862ef5d479aab5bdf0e974e55bc"
Oct 30 01:13:08.753849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.753825    4576 generic.go:356] "Generic (PLEG): container finished" podID="556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" containerID="27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31" exitCode=0
Oct 30 01:13:08.753976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.753846    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl" event={"ID":"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75","Type":"ContainerDied","Data":"27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31"}
Oct 30 01:13:08.753976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.753878    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"
Oct 30 01:13:08.753976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.753884    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl" event={"ID":"556fdf7e-9b7d-483b-b3b6-04fa78ca3b75","Type":"ContainerDied","Data":"d669cb6dee7d5708d4bdd49ef4defaca76d4572d2a6b9be6b0b7b38ab3f06e66"}
Oct 30 01:13:08.757953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.757928    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-credential-socket\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758034    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-envoy\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758071    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-certs\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758107    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-token\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758136    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-podinfo\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758187    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-credential-socket" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:08.758385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758222    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-kube-api-access-prnzg\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-kube-api-access-prnzg\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758371    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-socket\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758410    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-certs" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:08.758504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758419    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-data\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758451    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istiod-ca-cert\") pod \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\" (UID: \"dbc5c913-545b-43c0-ac8d-ab314d7ffb77\") "
Oct 30 01:13:08.758667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758632    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-socket" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:08.758847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.758804    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-data" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:08.759679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.759247    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.759679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.759274    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.759679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.759288    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.759679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.759301    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.759679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.759336    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istiod-ca-cert" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:08.760870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.760837    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-podinfo" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:08.760961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.760893    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-token" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:08.761010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.760978    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-envoy" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:08.762207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.762185    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-kube-api-access-prnzg" pod "dbc5c913-545b-43c0-ac8d-ab314d7ffb77" (UID: "dbc5c913-545b-43c0-ac8d-ab314d7ffb77"). InnerVolumeSpecName "kube-api-access-prnzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:08.764086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.764068    4576 scope.go:122] "RemoveContainer" containerID="b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25"
Oct 30 01:13:08.764398 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:08.764378    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25\": container with ID starting with b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25 not found: ID does not exist" containerID="b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25"
Oct 30 01:13:08.764485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.764405    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25"} err="failed to get container status \"b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25\": rpc error: code = NotFound desc = could not find container \"b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25\": container with ID starting with b36e88fc090a77b84c18266059fe2d8559e8ad26d76fccfc5c5d601075665f25 not found: ID does not exist"
Oct 30 01:13:08.764485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.764427    4576 scope.go:122] "RemoveContainer" containerID="27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31"
Oct 30 01:13:08.774905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.774888    4576 scope.go:122] "RemoveContainer" containerID="27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31"
Oct 30 01:13:08.775170 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:08.775151    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31\": container with ID starting with 27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31 not found: ID does not exist" containerID="27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31"
Oct 30 01:13:08.775243 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.775176    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31"} err="failed to get container status \"27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31\": rpc error: code = NotFound desc = could not find container \"27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31\": container with ID starting with 27589ff75eccf0f768cf70e88c883e1d66810baf1e0c9e311484a67530ad5e31 not found: ID does not exist"
Oct 30 01:13:08.833322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.833282    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"]
Oct 30 01:13:08.838274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.837895    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2bf8a-c6656d895-299fl"]
Oct 30 01:13:08.860133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.860098    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.860133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.860128    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.860133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.860137    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.860413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.860146    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.860413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.860174    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prnzg\" (UniqueName: \"kubernetes.io/projected/dbc5c913-545b-43c0-ac8d-ab314d7ffb77-kube-api-access-prnzg\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:08.950919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:08.950883    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:09.101057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.101013    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"]
Oct 30 01:13:09.108408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.108380    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-9kr4x"]
Oct 30 01:13:09.256435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.256409    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 01:13:09.367503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367447    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-podinfo\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.367700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367524    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-certs\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.367700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367630    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-kube-api-access-kssfg\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-kube-api-access-kssfg\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.367823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367707    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-data\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.367823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367763    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-credential-socket\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.367823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367818    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-socket\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.367979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367838    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-certs" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.367979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367929    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-token\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-token\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.367979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367943    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-data" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.368141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.367997    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istiod-ca-cert\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.368141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.368049    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-credential-socket" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.368141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.368063    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-envoy\") pod \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\" (UID: \"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15\") "
Oct 30 01:13:09.368141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.368084    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-socket" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.368581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.368450    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istiod-ca-cert" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:09.369041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.369014    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.369134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.369045    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.369134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.369062    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.369134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.369075    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.369134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.369088    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.370181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.370152    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-kube-api-access-kssfg" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "kube-api-access-kssfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:09.370318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.370269    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-podinfo" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:09.370409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.370392    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-token" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:09.370536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.370518    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-envoy" pod "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" (UID: "e3964518-d1f8-4c0b-ab7f-2ff3afec1e15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.470242 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.470192    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.470242 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.470224    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.470242 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.470250    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kssfg\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-kube-api-access-kssfg\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.470489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.470261    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.765122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.765040    4576 generic.go:356] "Generic (PLEG): container finished" podID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerID="dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15" exitCode=0
Oct 30 01:13:09.765122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.765096    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"
Oct 30 01:13:09.765389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.765121    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" event={"ID":"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15","Type":"ContainerDied","Data":"dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15"}
Oct 30 01:13:09.765389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.765186    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm" event={"ID":"e3964518-d1f8-4c0b-ab7f-2ff3afec1e15","Type":"ContainerDied","Data":"a275089fd437b52cdf4696c929833ef1c96480eae8a6981fb531eef390a9f8ee"}
Oct 30 01:13:09.765389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.765215    4576 scope.go:122] "RemoveContainer" containerID="dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15"
Oct 30 01:13:09.768695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.768518    4576 generic.go:356] "Generic (PLEG): container finished" podID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerID="18ed205a8bffeeba972efab4c7763351d6f56ff28264ec4943e1a1cc424eda36" exitCode=0
Oct 30 01:13:09.768695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.768561    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" event={"ID":"cea14f71-1fc9-426a-9ed4-8148160ff3aa","Type":"ContainerDied","Data":"18ed205a8bffeeba972efab4c7763351d6f56ff28264ec4943e1a1cc424eda36"}
Oct 30 01:13:09.778269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.778252    4576 scope.go:122] "RemoveContainer" containerID="dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15"
Oct 30 01:13:09.778525 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:09.778495    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15\": container with ID starting with dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15 not found: ID does not exist" containerID="dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15"
Oct 30 01:13:09.778616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.778523    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15"} err="failed to get container status \"dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15\": rpc error: code = NotFound desc = could not find container \"dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15\": container with ID starting with dc4bfac75fe314750610a6b8a2a6bbe33f073a3435d5295491db395c01953f15 not found: ID does not exist"
Oct 30 01:13:09.779162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.779147    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 01:13:09.874093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.873909    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"]
Oct 30 01:13:09.874462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.874437    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-kube-api-access-lbhzc\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-kube-api-access-lbhzc\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.874751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.874728    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istiod-ca-cert\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.874935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.874915    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-socket\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.875090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.875074    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-credential-socket\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.875311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.875293    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-envoy\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.875514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.875496    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-podinfo\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.875649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.875634    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-token\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-token\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.875834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.875805    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-certs\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.875948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.875858    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-data\") pod \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\" (UID: \"cea14f71-1fc9-426a-9ed4-8148160ff3aa\") "
Oct 30 01:13:09.877838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.875721    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istiod-ca-cert" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:09.877838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.876703    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-data" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.877838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.876878    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-socket" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.877838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.877038    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-credential-socket" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.878096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.878071    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-certs" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.880392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.880358    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-envoy" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:09.880392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.880374    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-podinfo" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:09.880539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.880399    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-token" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:09.880539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.880428    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-kube-api-access-lbhzc" pod "cea14f71-1fc9-426a-9ed4-8148160ff3aa" (UID: "cea14f71-1fc9-426a-9ed4-8148160ff3aa"). InnerVolumeSpecName "kube-api-access-lbhzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:09.880664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.880578    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-c4vdm"]
Oct 30 01:13:09.977700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977664    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.977700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977693    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.977700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977703    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.977700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977711    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.977959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977718    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.977959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977726    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbhzc\" (UniqueName: \"kubernetes.io/projected/cea14f71-1fc9-426a-9ed4-8148160ff3aa-kube-api-access-lbhzc\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.977959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977736    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea14f71-1fc9-426a-9ed4-8148160ff3aa-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.977959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977744    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.977959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:09.977751    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea14f71-1fc9-426a-9ed4-8148160ff3aa-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:10.109494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.109392    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" path="/var/lib/kubelet/pods/556fdf7e-9b7d-483b-b3b6-04fa78ca3b75/volumes"
Oct 30 01:13:10.110007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.109988    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" path="/var/lib/kubelet/pods/dbc5c913-545b-43c0-ac8d-ab314d7ffb77/volumes"
Oct 30 01:13:10.110528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.110510    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" path="/var/lib/kubelet/pods/e3964518-d1f8-4c0b-ab7f-2ff3afec1e15/volumes"
Oct 30 01:13:10.389448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.389332    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:10.780255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.779891    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6" event={"ID":"cea14f71-1fc9-426a-9ed4-8148160ff3aa","Type":"ContainerDied","Data":"6a13af3bce5a734ad3fb12ccf8de71ba729381d1d2ddc008dce5a6ba4b7890ec"}
Oct 30 01:13:10.780255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.779948    4576 scope.go:122] "RemoveContainer" containerID="18ed205a8bffeeba972efab4c7763351d6f56ff28264ec4943e1a1cc424eda36"
Oct 30 01:13:10.780255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.780035    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"
Oct 30 01:13:10.868255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.866992    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"]
Oct 30 01:13:10.905943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.905912    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-t96g6"]
Oct 30 01:13:10.907210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:10.907116    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:11.099573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:11.099471    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:11.242199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:11.242153    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:11.318628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:11.318596    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"]
Oct 30 01:13:11.322303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:11.322223    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0680fa-4jwt6"]
Oct 30 01:13:11.810549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:11.810515    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:11.967630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:11.967598    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:12.079854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:12.079760    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:12.082243 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:12.082209    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"]
Oct 30 01:13:12.094331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:12.094299    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b2bf8a-8687k"]
Oct 30 01:13:12.110415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:12.110383    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11a240f-abdd-473c-9482-7893fcb3857e" path="/var/lib/kubelet/pods/c11a240f-abdd-473c-9482-7893fcb3857e/volumes"
Oct 30 01:13:12.111065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:12.111049    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" path="/var/lib/kubelet/pods/cea14f71-1fc9-426a-9ed4-8148160ff3aa/volumes"
Oct 30 01:13:12.111639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:12.111623    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25eaa9b-c082-482d-af2f-d353745e9d82" path="/var/lib/kubelet/pods/e25eaa9b-c082-482d-af2f-d353745e9d82/volumes"
Oct 30 01:13:12.193000 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:12.192971    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:12.578582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:12.578535    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:13.900317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:13.900283    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:14.613892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:14.613862    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:15.464978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.464944    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"]
Oct 30 01:13:15.465586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465571    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed" containerName="revoke-keys"
Oct 30 01:13:15.465631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465588    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed" containerName="revoke-keys"
Oct 30 01:13:15.465631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465614    4576 cpu_manager.go:414] "containerMap: removing container" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy"
Oct 30 01:13:15.465631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465619    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy"
Oct 30 01:13:15.465715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465631    4576 cpu_manager.go:414] "containerMap: removing container" podUID="556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" containerName="payload-pre-processing"
Oct 30 01:13:15.465715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465637    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" containerName="payload-pre-processing"
Oct 30 01:13:15.465715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465646    4576 cpu_manager.go:414] "containerMap: removing container" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy"
Oct 30 01:13:15.465715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465651    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy"
Oct 30 01:13:15.465715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465663    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy"
Oct 30 01:13:15.465715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465668    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy"
Oct 30 01:13:15.465884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465753    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed" containerName="revoke-keys"
Oct 30 01:13:15.465884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465765    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea14f71-1fc9-426a-9ed4-8148160ff3aa" containerName="istio-proxy"
Oct 30 01:13:15.465884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465772    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="556fdf7e-9b7d-483b-b3b6-04fa78ca3b75" containerName="payload-pre-processing"
Oct 30 01:13:15.465884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465779    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc5c913-545b-43c0-ac8d-ab314d7ffb77" containerName="istio-proxy"
Oct 30 01:13:15.465884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465785    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="228a0ad5-07f7-43ba-bc33-a2a61a97996f" containerName="istio-proxy"
Oct 30 01:13:15.465884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.465792    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3964518-d1f8-4c0b-ab7f-2ff3afec1e15" containerName="istio-proxy"
Oct 30 01:13:15.470971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.470949    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.485729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.485703    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"]
Oct 30 01:13:15.495634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.495613    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:15.495742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.495664    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:13:15.570900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.570863    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.571120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.570933    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.571202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.571114    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzsb\" (UniqueName: \"kubernetes.io/projected/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-kube-api-access-fdzsb\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.600107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.600073    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:15.673209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.672098    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzsb\" (UniqueName: \"kubernetes.io/projected/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-kube-api-access-fdzsb\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.673209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.672349    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.673209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.672401    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.673209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.673143    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.675089 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.675066    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.693741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.693670    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzsb\" (UniqueName: \"kubernetes.io/projected/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-kube-api-access-fdzsb\") pod \"maas-api-revoke-sub-4e12a61a-sps94\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:15.785504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:15.785411    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:16.089710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:16.089681    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"]
Oct 30 01:13:16.090790 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:16.090762    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c8ee45_2b1e_4f16_ac5c_ddf0e8cfe2f9.slice/crio-a48a086bba6da272f7b0dea6dab5212655fcbd896887411fabc3dda64d470a99 WatchSource:0}: Error finding container a48a086bba6da272f7b0dea6dab5212655fcbd896887411fabc3dda64d470a99: Status 404 returned error can't find the container with id a48a086bba6da272f7b0dea6dab5212655fcbd896887411fabc3dda64d470a99
Oct 30 01:13:16.864075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:16.864033    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94" event={"ID":"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9","Type":"ContainerStarted","Data":"cfc7ef71b733b52165afa5a8bbd0857efa390d68205b4bc0952fcc8a4c5f6605"}
Oct 30 01:13:16.864493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:16.864092    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94" event={"ID":"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9","Type":"ContainerStarted","Data":"a48a086bba6da272f7b0dea6dab5212655fcbd896887411fabc3dda64d470a99"}
Oct 30 01:13:16.908441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:16.908391    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94" podStartSLOduration=1.90837549 podStartE2EDuration="1.90837549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:16.907430257 +0000 UTC m=+2076.978754091" watchObservedRunningTime="2026-10-30 01:13:16.90837549 +0000 UTC m=+2076.979699326"
Oct 30 01:13:17.369917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:17.369883    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:17.869295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:17.869257    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:17.874180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:17.874149    4576 generic.go:356] "Generic (PLEG): container finished" podID="e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9" containerID="cfc7ef71b733b52165afa5a8bbd0857efa390d68205b4bc0952fcc8a4c5f6605" exitCode=0
Oct 30 01:13:17.874328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:17.874202    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94" event={"ID":"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9","Type":"ContainerDied","Data":"cfc7ef71b733b52165afa5a8bbd0857efa390d68205b4bc0952fcc8a4c5f6605"}
Oct 30 01:13:17.984765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:17.984733    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:18.146066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.145986    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"]
Oct 30 01:13:18.146540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.146506    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy" containerID="cri-o://4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b" gracePeriod=30
Oct 30 01:13:18.188970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.188913    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:18.424551 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:18.424470    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:13:18.674668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.674589    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"]
Oct 30 01:13:18.678645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.678627    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.696671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.696646    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"]
Oct 30 01:13:18.776439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.776411    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:18.839988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.839946    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.840176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.840094    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92pr2\" (UniqueName: \"kubernetes.io/projected/2eeafc9e-0d86-4c0f-ae49-fb8347223055-kube-api-access-92pr2\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.840294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.840213    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.900169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.900133    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:18.942052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.942017    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.942327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.942253    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.942438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.942338    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92pr2\" (UniqueName: \"kubernetes.io/projected/2eeafc9e-0d86-4c0f-ae49-fb8347223055-kube-api-access-92pr2\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.943084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.943058    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.967791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.967757    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.970828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.970798    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92pr2\" (UniqueName: \"kubernetes.io/projected/2eeafc9e-0d86-4c0f-ae49-fb8347223055-kube-api-access-92pr2\") pod \"maas-api-revoke-keys-e2e-inf-b-d54608-flq7x\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:18.996800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:18.996769    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:19.055742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.055709    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"]
Oct 30 01:13:19.056077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.056028    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy" containerID="cri-o://0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d" gracePeriod=30
Oct 30 01:13:19.092161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.092091    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:19.117620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.117324    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:19.207340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.207150    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:19.257957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.257909    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-openshift-ca\") pod \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") "
Oct 30 01:13:19.258153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.258024    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-kube-api-access-fdzsb\" (UniqueName: \"kubernetes.io/projected/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-kube-api-access-fdzsb\") pod \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") "
Oct 30 01:13:19.258224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.258187    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-xks-ca\") pod \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\" (UID: \"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9\") "
Oct 30 01:13:19.258682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.258364    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-openshift-ca" pod "e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9" (UID: "e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:19.258999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.258930    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:19.261310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.261281    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-xks-ca" pod "e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9" (UID: "e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:19.261415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.261317    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-kube-api-access-fdzsb" pod "e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9" (UID: "e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9"). InnerVolumeSpecName "kube-api-access-fdzsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:19.360076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.360042    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdzsb\" (UniqueName: \"kubernetes.io/projected/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-kube-api-access-fdzsb\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:19.360076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.360078    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:19.381846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.381822    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"]
Oct 30 01:13:19.383078 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:19.383048    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eeafc9e_0d86_4c0f_ae49_fb8347223055.slice/crio-49011d2c1faa6cc11884d4e15b2bb2f63d9ff477d5a68cb447e28041b83aa223 WatchSource:0}: Error finding container 49011d2c1faa6cc11884d4e15b2bb2f63d9ff477d5a68cb447e28041b83aa223: Status 404 returned error can't find the container with id 49011d2c1faa6cc11884d4e15b2bb2f63d9ff477d5a68cb447e28041b83aa223
Oct 30 01:13:19.890395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.890306    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x" event={"ID":"2eeafc9e-0d86-4c0f-ae49-fb8347223055","Type":"ContainerStarted","Data":"37fadb3db577e51d130cbef94f91da2c5fbb405f967560aeb535da3e429d351a"}
Oct 30 01:13:19.890395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.890369    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x" event={"ID":"2eeafc9e-0d86-4c0f-ae49-fb8347223055","Type":"ContainerStarted","Data":"49011d2c1faa6cc11884d4e15b2bb2f63d9ff477d5a68cb447e28041b83aa223"}
Oct 30 01:13:19.891950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.891921    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"
Oct 30 01:13:19.892077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.891963    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94" event={"ID":"e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9","Type":"ContainerDied","Data":"a48a086bba6da272f7b0dea6dab5212655fcbd896887411fabc3dda64d470a99"}
Oct 30 01:13:19.892077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.892003    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a48a086bba6da272f7b0dea6dab5212655fcbd896887411fabc3dda64d470a99"
Oct 30 01:13:19.932102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:19.932044    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x" podStartSLOduration=1.932029102 podStartE2EDuration="1.932029102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:19.931310029 +0000 UTC m=+2080.002633864" watchObservedRunningTime="2026-10-30 01:13:19.932029102 +0000 UTC m=+2080.003352937"
Oct 30 01:13:20.318059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:20.318026    4576 patch_prober.go:29] interesting pod/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:13:20.318202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:20.318079    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:13:20.900679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:20.900645    4576 generic.go:356] "Generic (PLEG): container finished" podID="2eeafc9e-0d86-4c0f-ae49-fb8347223055" containerID="37fadb3db577e51d130cbef94f91da2c5fbb405f967560aeb535da3e429d351a" exitCode=0
Oct 30 01:13:20.900844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:20.900737    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x" event={"ID":"2eeafc9e-0d86-4c0f-ae49-fb8347223055","Type":"ContainerDied","Data":"37fadb3db577e51d130cbef94f91da2c5fbb405f967560aeb535da3e429d351a"}
Oct 30 01:13:21.162913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:21.162824    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:22.085080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.085056    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:22.094547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.094519    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-xks-ca\") pod \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") "
Oct 30 01:13:22.094670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.094570    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eeafc9e-0d86-4c0f-ae49-fb8347223055-kube-api-access-92pr2\" (UniqueName: \"kubernetes.io/projected/2eeafc9e-0d86-4c0f-ae49-fb8347223055-kube-api-access-92pr2\") pod \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") "
Oct 30 01:13:22.094670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.094654    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-openshift-ca\") pod \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\" (UID: \"2eeafc9e-0d86-4c0f-ae49-fb8347223055\") "
Oct 30 01:13:22.095032 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.095004    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-openshift-ca" pod "2eeafc9e-0d86-4c0f-ae49-fb8347223055" (UID: "2eeafc9e-0d86-4c0f-ae49-fb8347223055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:22.095363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.095344    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:22.096948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.096925    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-xks-ca" pod "2eeafc9e-0d86-4c0f-ae49-fb8347223055" (UID: "2eeafc9e-0d86-4c0f-ae49-fb8347223055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:22.097166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.097142    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eeafc9e-0d86-4c0f-ae49-fb8347223055-kube-api-access-92pr2" pod "2eeafc9e-0d86-4c0f-ae49-fb8347223055" (UID: "2eeafc9e-0d86-4c0f-ae49-fb8347223055"). InnerVolumeSpecName "kube-api-access-92pr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:22.197105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.197076    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eeafc9e-0d86-4c0f-ae49-fb8347223055-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:22.197105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.197106    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92pr2\" (UniqueName: \"kubernetes.io/projected/2eeafc9e-0d86-4c0f-ae49-fb8347223055-kube-api-access-92pr2\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:22.874550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.874512    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:22.929613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.929575    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x" event={"ID":"2eeafc9e-0d86-4c0f-ae49-fb8347223055","Type":"ContainerDied","Data":"49011d2c1faa6cc11884d4e15b2bb2f63d9ff477d5a68cb447e28041b83aa223"}
Oct 30 01:13:22.929825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.929667    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49011d2c1faa6cc11884d4e15b2bb2f63d9ff477d5a68cb447e28041b83aa223"
Oct 30 01:13:22.929825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:22.929743    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"
Oct 30 01:13:23.430000 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.429905    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:13:23.513098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513066    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-data\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.513351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513126    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-certs\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.513351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513166    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-credential-socket\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.513351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513272    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-socket\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.513351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513313    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-token\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-token\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.513351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513347    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-kube-api-access-9x5pq\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-kube-api-access-9x5pq\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.513667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513415    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-data" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:23.513667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513472    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-envoy\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.513667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513498    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-podinfo\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.513667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513531    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-certs" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:23.513667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.513581    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6000abc5-dcfc-4c59-a169-037cfb59be1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6000abc5-dcfc-4c59-a169-037cfb59be1b-istiod-ca-cert\") pod \"6000abc5-dcfc-4c59-a169-037cfb59be1b\" (UID: \"6000abc5-dcfc-4c59-a169-037cfb59be1b\") "
Oct 30 01:13:23.514200 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.514166    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6000abc5-dcfc-4c59-a169-037cfb59be1b-istiod-ca-cert" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:23.514547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.514523    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-socket" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:23.514789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.514754    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-credential-socket" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:23.514888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.514803    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.514888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.514821    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.514888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.514835    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.514888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.514848    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6000abc5-dcfc-4c59-a169-037cfb59be1b-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.516210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.516184    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-envoy" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:23.516570 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.516537    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-kube-api-access-9x5pq" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "kube-api-access-9x5pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:23.516869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.516842    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-token" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:23.517173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.517155    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-podinfo" pod "6000abc5-dcfc-4c59-a169-037cfb59be1b" (UID: "6000abc5-dcfc-4c59-a169-037cfb59be1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:23.616371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.616343    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.616371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.616367    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.616371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.616378    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x5pq\" (UniqueName: \"kubernetes.io/projected/6000abc5-dcfc-4c59-a169-037cfb59be1b-kube-api-access-9x5pq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.616613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.616388    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.616613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.616397    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6000abc5-dcfc-4c59-a169-037cfb59be1b-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.939143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.939110    4576 generic.go:356] "Generic (PLEG): container finished" podID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerID="4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b" exitCode=0
Oct 30 01:13:23.939383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.939161    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"
Oct 30 01:13:23.939383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.939198    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" event={"ID":"6000abc5-dcfc-4c59-a169-037cfb59be1b","Type":"ContainerDied","Data":"4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b"}
Oct 30 01:13:23.939383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.939286    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r" event={"ID":"6000abc5-dcfc-4c59-a169-037cfb59be1b","Type":"ContainerDied","Data":"e2e53286c1891cc6a2179af69330dc82034bc215e180fce3f2703e1568eefbec"}
Oct 30 01:13:23.939383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.939324    4576 scope.go:122] "RemoveContainer" containerID="4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b"
Oct 30 01:13:23.951807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.951786    4576 scope.go:122] "RemoveContainer" containerID="4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b"
Oct 30 01:13:23.952116 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:23.952093    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b\": container with ID starting with 4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b not found: ID does not exist" containerID="4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b"
Oct 30 01:13:23.952165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.952130    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b"} err="failed to get container status \"4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b\": rpc error: code = NotFound desc = could not find container \"4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b\": container with ID starting with 4bf7b0561377939760c2696bbd94dc70dff17e4a94837a4089c41070eecb0b1b not found: ID does not exist"
Oct 30 01:13:23.998777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:23.998742    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"]
Oct 30 01:13:24.010661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.010636    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0680fa-openshift-default-8545d54d69-kql7r"]
Oct 30 01:13:24.111809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.111775    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" path="/var/lib/kubelet/pods/6000abc5-dcfc-4c59-a169-037cfb59be1b/volumes"
Oct 30 01:13:24.305267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.302662    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:24.345941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.345910    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:13:24.374110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374081    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"]
Oct 30 01:13:24.374691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374675    4576 cpu_manager.go:414] "containerMap: removing container" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy"
Oct 30 01:13:24.374752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374692    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy"
Oct 30 01:13:24.374752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374709    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2eeafc9e-0d86-4c0f-ae49-fb8347223055" containerName="revoke-keys"
Oct 30 01:13:24.374752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374714    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eeafc9e-0d86-4c0f-ae49-fb8347223055" containerName="revoke-keys"
Oct 30 01:13:24.374752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374724    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy"
Oct 30 01:13:24.374752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374729    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy"
Oct 30 01:13:24.374752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374741    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9" containerName="revoke-keys"
Oct 30 01:13:24.374752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374745    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9" containerName="revoke-keys"
Oct 30 01:13:24.375033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374757    4576 cpu_manager.go:414] "containerMap: removing container" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy"
Oct 30 01:13:24.375033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374761    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy"
Oct 30 01:13:24.375033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374876    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy"
Oct 30 01:13:24.375033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374885    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eeafc9e-0d86-4c0f-ae49-fb8347223055" containerName="revoke-keys"
Oct 30 01:13:24.375033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374892    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6000abc5-dcfc-4c59-a169-037cfb59be1b" containerName="istio-proxy"
Oct 30 01:13:24.375033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.374900    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9" containerName="revoke-keys"
Oct 30 01:13:24.378408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.378378    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.395738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.395529    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-95fe33\"/\"kube-root-ca.crt\""
Oct 30 01:13:24.395738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.395664    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-95fe33\"/\"openshift-service-ca.crt\""
Oct 30 01:13:24.402440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.402375    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"]
Oct 30 01:13:24.404517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.404495    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-95fe33\"/\"default-dockercfg-274qz\""
Oct 30 01:13:24.404640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.404555    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-95fe33\"/\"test-model-95fe33-kserve-self-signed-certs\""
Oct 30 01:13:24.426512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.426475    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-credential-socket\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.426695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.426519    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-envoy\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.426695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.426571    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-data\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.426831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.426711    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istiod-ca-cert\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.426831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.426749    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-credential-socket" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:24.426831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.426803    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-podinfo\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.427003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.426874    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-data" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:24.427003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.426902    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-socket\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.427164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.427144    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-socket" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:24.427249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.427142    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istiod-ca-cert" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:24.427249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.427198    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-certs\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.427378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.427300    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-token\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-token\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.427529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.427504    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-certs" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:24.427645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.427628    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-kube-api-access-klj7j\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-kube-api-access-klj7j\") pod \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\" (UID: \"a890833d-89d5-40a9-a140-5e49fb5d4d7b\") "
Oct 30 01:13:24.428176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428148    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hkv\" (UniqueName: \"kubernetes.io/projected/f981d537-1da3-417f-b726-b1d04056884a-kube-api-access-c2hkv\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.428409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428298    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-home\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.428409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428360    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-tmp-dir\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.428655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428639    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-model-cache\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.428756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428680    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981d537-1da3-417f-b726-b1d04056884a-tls-certs\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.428756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428711    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-dshm\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.428964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428938    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.429086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428967    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.429086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428981    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.429086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.428992    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.429086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.429005    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.430219 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.430192    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-envoy" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:24.430636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.430327    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-token" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:24.430636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.430458    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-kube-api-access-klj7j" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "kube-api-access-klj7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:24.431558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.431531    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-podinfo" pod "a890833d-89d5-40a9-a140-5e49fb5d4d7b" (UID: "a890833d-89d5-40a9-a140-5e49fb5d4d7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:13:24.452504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.452202    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:24.530555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530462    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-model-cache\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.530555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530524    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981d537-1da3-417f-b726-b1d04056884a-tls-certs\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.530856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530557    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-dshm\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.530856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530712    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hkv\" (UniqueName: \"kubernetes.io/projected/f981d537-1da3-417f-b726-b1d04056884a-kube-api-access-c2hkv\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.530856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530766    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-home\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.530856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530821    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-tmp-dir\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.531083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530940    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.531083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530958    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klj7j\" (UniqueName: \"kubernetes.io/projected/a890833d-89d5-40a9-a140-5e49fb5d4d7b-kube-api-access-klj7j\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.531083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530974    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.531083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530977    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-model-cache\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.531083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.530989    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a890833d-89d5-40a9-a140-5e49fb5d4d7b-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.531410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.531274    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-tmp-dir\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.531723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.531699    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-home\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.533285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.533226    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-dshm\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.533455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.533435    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981d537-1da3-417f-b726-b1d04056884a-tls-certs\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.555346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.555312    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hkv\" (UniqueName: \"kubernetes.io/projected/f981d537-1da3-417f-b726-b1d04056884a-kube-api-access-c2hkv\") pod \"test-model-95fe33-kserve-544ccdf65-twrwd\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") " pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.706587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.706546    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:24.947775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.947742    4576 generic.go:356] "Generic (PLEG): container finished" podID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerID="0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d" exitCode=0
Oct 30 01:13:24.947958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.947797    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"
Oct 30 01:13:24.947958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.947828    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" event={"ID":"a890833d-89d5-40a9-a140-5e49fb5d4d7b","Type":"ContainerDied","Data":"0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d"}
Oct 30 01:13:24.947958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.947887    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7" event={"ID":"a890833d-89d5-40a9-a140-5e49fb5d4d7b","Type":"ContainerDied","Data":"b44538a009b7314f131cdac6a12bea6782aaa3e0914e731dc3424eb9570c1934"}
Oct 30 01:13:24.947958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.947911    4576 scope.go:122] "RemoveContainer" containerID="0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d"
Oct 30 01:13:24.958819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.958776    4576 scope.go:122] "RemoveContainer" containerID="e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315"
Oct 30 01:13:24.968541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.968512    4576 scope.go:122] "RemoveContainer" containerID="0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d"
Oct 30 01:13:24.968851 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:24.968827    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d\": container with ID starting with 0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d not found: ID does not exist" containerID="0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d"
Oct 30 01:13:24.968916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.968861    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d"} err="failed to get container status \"0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d\": rpc error: code = NotFound desc = could not find container \"0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d\": container with ID starting with 0aed85877e966d1b37ebfbfaa92ef6753cbf7478f83e4547ef707d16e501349d not found: ID does not exist"
Oct 30 01:13:24.968916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.968885    4576 scope.go:122] "RemoveContainer" containerID="e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315"
Oct 30 01:13:24.969161 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:24.969135    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315\": container with ID starting with e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315 not found: ID does not exist" containerID="e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315"
Oct 30 01:13:24.969213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:24.969169    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315"} err="failed to get container status \"e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315\": rpc error: code = NotFound desc = could not find container \"e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315\": container with ID starting with e94b5489b7173637050296c66b63b51e65442a87861339cd4b5acd00c1680315 not found: ID does not exist"
Oct 30 01:13:25.007611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.007583    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"]
Oct 30 01:13:25.010822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.010798    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b2bf8a-openshift-default-7d8c6f557f-lknx7"]
Oct 30 01:13:25.021851 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:25.021820    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf981d537_1da3_417f_b726_b1d04056884a.slice/crio-ea607f94fb95c8443b57922d83c3e9117380f97ff9939aa89392ff7da9ac66b6 WatchSource:0}: Error finding container ea607f94fb95c8443b57922d83c3e9117380f97ff9939aa89392ff7da9ac66b6: Status 404 returned error can't find the container with id ea607f94fb95c8443b57922d83c3e9117380f97ff9939aa89392ff7da9ac66b6
Oct 30 01:13:25.022247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.022206    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"]
Oct 30 01:13:25.167585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.167551    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"]
Oct 30 01:13:25.168295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.168274    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" containerName="istio-proxy"
Oct 30 01:13:25.171988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.171970    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.189667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.189633    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-283eb6-openshift-default-dockercfg-kw2jr\""
Oct 30 01:13:25.194000 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.193973    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"]
Oct 30 01:13:25.244405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244312    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bc6efc-6a25-4023-be95-ffb996da747c-istiod-ca-cert\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.244588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244449    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.244588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244496    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-istio-token\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.244588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244550    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-credential-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.244764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244634    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-envoy\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.244764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244664    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6hk\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-kube-api-access-rb6hk\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.244764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244715    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-data\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.244924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244810    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bc6efc-6a25-4023-be95-ffb996da747c-istio-podinfo\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.244924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.244882    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-certs\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.346425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346392    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-data\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.346627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346452    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bc6efc-6a25-4023-be95-ffb996da747c-istio-podinfo\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.346627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346549    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-certs\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.346749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346642    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bc6efc-6a25-4023-be95-ffb996da747c-istiod-ca-cert\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.346749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346707    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.346862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346752    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-istio-token\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.346862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346807    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-credential-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.346862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346856    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-envoy\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.347016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346872    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-data\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.347307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.347279    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.347576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.347556    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-credential-socket\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.347703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.347674    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bc6efc-6a25-4023-be95-ffb996da747c-istiod-ca-cert\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.347774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.346880    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6hk\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-kube-api-access-rb6hk\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.348156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.348122    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-certs\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.350050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.350027    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bc6efc-6a25-4023-be95-ffb996da747c-istio-podinfo\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.350913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.350884    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-envoy\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.370261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.368987    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6hk\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-kube-api-access-rb6hk\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.370261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.369384    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-istio-token\") pod \"e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") " pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.505926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.505834    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:25.834562 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.834532    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"]
Oct 30 01:13:25.835371 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:25.835346    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bc6efc_6a25_4023_be95_ffb996da747c.slice/crio-2564d8a5336775d67a7b57bdf54dd838a5c6e183be0da89f1cca119a7706f718 WatchSource:0}: Error finding container 2564d8a5336775d67a7b57bdf54dd838a5c6e183be0da89f1cca119a7706f718: Status 404 returned error can't find the container with id 2564d8a5336775d67a7b57bdf54dd838a5c6e183be0da89f1cca119a7706f718
Oct 30 01:13:25.838347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.838312    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:13:25.838444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.838377    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:13:25.838444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.838406    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:13:25.873465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.873436    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:25.958662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.958618    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" event={"ID":"13bc6efc-6a25-4023-be95-ffb996da747c","Type":"ContainerStarted","Data":"ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd"}
Oct 30 01:13:25.958862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.958680    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" event={"ID":"13bc6efc-6a25-4023-be95-ffb996da747c","Type":"ContainerStarted","Data":"2564d8a5336775d67a7b57bdf54dd838a5c6e183be0da89f1cca119a7706f718"}
Oct 30 01:13:25.961916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.961887    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd" event={"ID":"f981d537-1da3-417f-b726-b1d04056884a","Type":"ContainerStarted","Data":"a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012"}
Oct 30 01:13:25.961916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:25.961918    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd" event={"ID":"f981d537-1da3-417f-b726-b1d04056884a","Type":"ContainerStarted","Data":"ea607f94fb95c8443b57922d83c3e9117380f97ff9939aa89392ff7da9ac66b6"}
Oct 30 01:13:26.004787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:26.004730    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podStartSLOduration=1.004715195 podStartE2EDuration="1.004715195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:26.002245412 +0000 UTC m=+2086.073569245" watchObservedRunningTime="2026-10-30 01:13:26.004715195 +0000 UTC m=+2086.076039095"
Oct 30 01:13:26.110914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:26.110820    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a890833d-89d5-40a9-a140-5e49fb5d4d7b" path="/var/lib/kubelet/pods/a890833d-89d5-40a9-a140-5e49fb5d4d7b/volumes"
Oct 30 01:13:26.506065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:26.506027    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:26.507864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:26.507838    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:13:26.507991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:26.507885    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 01:13:26.707402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:26.707371    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:27.456050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:27.456014    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:27.507981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:27.507940    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:13:27.508511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:27.507992    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 01:13:27.999501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:27.999437    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd" podStartSLOduration=3.999417267 podStartE2EDuration="3.999417267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:26.046423719 +0000 UTC m=+2086.117747566" watchObservedRunningTime="2026-10-30 01:13:27.999417267 +0000 UTC m=+2088.070741114"
Oct 30 01:13:28.010259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.007225    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"]
Oct 30 01:13:28.036424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.036395    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:28.498308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.497588    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"]
Oct 30 01:13:28.502569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.502540    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.506493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.506336    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:13:28.506493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.506386    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 01:13:28.520303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.512770    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"]
Oct 30 01:13:28.520303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.518774    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:13:28.520303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.518779    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:28.520303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.519920    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.520303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.519972    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg2p4\" (UniqueName: \"kubernetes.io/projected/f2ca165b-9d10-4098-b61f-698e2669f0ac-kube-api-access-wg2p4\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.520303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.520023    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.621307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.621269    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.621510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.621320    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg2p4\" (UniqueName: \"kubernetes.io/projected/f2ca165b-9d10-4098-b61f-698e2669f0ac-kube-api-access-wg2p4\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.621510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.621364    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.622278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.622248    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.623998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.623965    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.642096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.642067    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg2p4\" (UniqueName: \"kubernetes.io/projected/f2ca165b-9d10-4098-b61f-698e2669f0ac-kube-api-access-wg2p4\") pod \"maas-api-revoke-sub-4d637624-shg29\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.823859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.823781    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:28.965776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:28.965737    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:29.159426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:29.159396    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"]
Oct 30 01:13:29.159858 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:29.159836    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ca165b_9d10_4098_b61f_698e2669f0ac.slice/crio-94e1a62f518f56d9a149fe7682e1f2bfc1f0ae8a22bd8d5cc634d71f432b747e WatchSource:0}: Error finding container 94e1a62f518f56d9a149fe7682e1f2bfc1f0ae8a22bd8d5cc634d71f432b747e: Status 404 returned error can't find the container with id 94e1a62f518f56d9a149fe7682e1f2bfc1f0ae8a22bd8d5cc634d71f432b747e
Oct 30 01:13:29.507860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:29.507817    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:13:29.508075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:29.507898    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 01:13:29.555317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:29.555279    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"]
Oct 30 01:13:29.705079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:29.705019    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:29.812761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:29.812662    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:30.039125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:30.039087    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29" event={"ID":"f2ca165b-9d10-4098-b61f-698e2669f0ac","Type":"ContainerStarted","Data":"01b8c4ce399bca69e38bf73c16fc28d9e2335661c17a27a9b7c2f4d8d46df80d"}
Oct 30 01:13:30.039295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:30.039139    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29" event={"ID":"f2ca165b-9d10-4098-b61f-698e2669f0ac","Type":"ContainerStarted","Data":"94e1a62f518f56d9a149fe7682e1f2bfc1f0ae8a22bd8d5cc634d71f432b747e"}
Oct 30 01:13:30.108108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:30.107978    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29" podStartSLOduration=2.10796163 podStartE2EDuration="2.10796163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:30.107289257 +0000 UTC m=+2090.178613094" watchObservedRunningTime="2026-10-30 01:13:30.10796163 +0000 UTC m=+2090.179285465"
Oct 30 01:13:30.506436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:30.506400    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:13:30.506638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:30.506463    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 01:13:30.764206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:30.764119    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:31.051469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:31.048948    4576 generic.go:356] "Generic (PLEG): container finished" podID="f2ca165b-9d10-4098-b61f-698e2669f0ac" containerID="01b8c4ce399bca69e38bf73c16fc28d9e2335661c17a27a9b7c2f4d8d46df80d" exitCode=0
Oct 30 01:13:31.051469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:31.049023    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29" event={"ID":"f2ca165b-9d10-4098-b61f-698e2669f0ac","Type":"ContainerDied","Data":"01b8c4ce399bca69e38bf73c16fc28d9e2335661c17a27a9b7c2f4d8d46df80d"}
Oct 30 01:13:31.507309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:31.507274    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 01:13:31.507494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:31.507346    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 01:13:32.234686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.234654    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:32.301379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.293733    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ca165b-9d10-4098-b61f-698e2669f0ac-kube-api-access-wg2p4\" (UniqueName: \"kubernetes.io/projected/f2ca165b-9d10-4098-b61f-698e2669f0ac-kube-api-access-wg2p4\") pod \"f2ca165b-9d10-4098-b61f-698e2669f0ac\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") "
Oct 30 01:13:32.301379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.293834    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-xks-ca\") pod \"f2ca165b-9d10-4098-b61f-698e2669f0ac\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") "
Oct 30 01:13:32.301379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.293895    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-openshift-ca\") pod \"f2ca165b-9d10-4098-b61f-698e2669f0ac\" (UID: \"f2ca165b-9d10-4098-b61f-698e2669f0ac\") "
Oct 30 01:13:32.301379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.300256    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-openshift-ca" pod "f2ca165b-9d10-4098-b61f-698e2669f0ac" (UID: "f2ca165b-9d10-4098-b61f-698e2669f0ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:32.314408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.306950    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ca165b-9d10-4098-b61f-698e2669f0ac-kube-api-access-wg2p4" pod "f2ca165b-9d10-4098-b61f-698e2669f0ac" (UID: "f2ca165b-9d10-4098-b61f-698e2669f0ac"). InnerVolumeSpecName "kube-api-access-wg2p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:32.314408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.308372    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-xks-ca" pod "f2ca165b-9d10-4098-b61f-698e2669f0ac" (UID: "f2ca165b-9d10-4098-b61f-698e2669f0ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:32.360724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.360689    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:32.396340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.396306    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg2p4\" (UniqueName: \"kubernetes.io/projected/f2ca165b-9d10-4098-b61f-698e2669f0ac-kube-api-access-wg2p4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:32.396526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.396446    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:32.396526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.396489    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ca165b-9d10-4098-b61f-698e2669f0ac-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:32.541666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.541632    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:32.542879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.542136    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:32.543257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.543222    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:13:32.737008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:32.736967    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:33.069550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:33.069468    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"
Oct 30 01:13:33.069550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:33.069477    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29" event={"ID":"f2ca165b-9d10-4098-b61f-698e2669f0ac","Type":"ContainerDied","Data":"94e1a62f518f56d9a149fe7682e1f2bfc1f0ae8a22bd8d5cc634d71f432b747e"}
Oct 30 01:13:33.069745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:33.069562    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e1a62f518f56d9a149fe7682e1f2bfc1f0ae8a22bd8d5cc634d71f432b747e"
Oct 30 01:13:33.999303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:33.999267    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:34.556038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:34.555973    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" podUID="a475c6d4-042d-45cd-850e-49d0eda403c5" containerName="payload-pre-processing" containerID="cri-o://adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7" gracePeriod=25
Oct 30 01:13:34.596918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:34.596882    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" podUID="a475c6d4-042d-45cd-850e-49d0eda403c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:13:34.707721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:34.707692    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:34.720199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:34.720169    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:35.096056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:35.096029    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:13:35.551346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:35.551315    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:35.654632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:35.654601    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:37.101578 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:37.101548    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:37.208792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:37.208756    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:38.560130 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:38.560092    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:39.621818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:39.621776    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:40.195280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:40.195210    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:40.301888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:40.301851    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:40.411873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:40.411835    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:41.151572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.151492    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"]
Oct 30 01:13:41.207166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.207134    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:41.347283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.346006    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"]
Oct 30 01:13:41.347283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.346914    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f2ca165b-9d10-4098-b61f-698e2669f0ac" containerName="revoke-keys"
Oct 30 01:13:41.347283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.346935    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ca165b-9d10-4098-b61f-698e2669f0ac" containerName="revoke-keys"
Oct 30 01:13:41.347283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.347150    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ca165b-9d10-4098-b61f-698e2669f0ac" containerName="revoke-keys"
Oct 30 01:13:41.355485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.355461    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.363139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.363092    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"]
Oct 30 01:13:41.373449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.373421    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:13:41.373597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.373512    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:41.436666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.436582    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcc6\" (UniqueName: \"kubernetes.io/projected/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-kube-api-access-wfcc6\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.436809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.436758    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.436899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.436886    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.538554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.538522    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.538750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.538645    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.538750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.538739    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcc6\" (UniqueName: \"kubernetes.io/projected/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-kube-api-access-wfcc6\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.539531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.539504    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.541441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.541413    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.560337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.560307    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcc6\" (UniqueName: \"kubernetes.io/projected/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-kube-api-access-wfcc6\") pod \"maas-api-revoke-keys-e2e-inf-a-286d36-zv884\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.678181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.678142    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:41.894876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.894846    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:41.982413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:41.982385    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"]
Oct 30 01:13:41.983628 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:41.983602    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868ab4a6_9d55_4acf_8ef0_63c8a5c626f8.slice/crio-c21f54f0389f5dd90f44f9a04655fcdedd7d8f195eb31ae6ccbfac7c7cab8a66 WatchSource:0}: Error finding container c21f54f0389f5dd90f44f9a04655fcdedd7d8f195eb31ae6ccbfac7c7cab8a66: Status 404 returned error can't find the container with id c21f54f0389f5dd90f44f9a04655fcdedd7d8f195eb31ae6ccbfac7c7cab8a66
Oct 30 01:13:42.177639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:42.177605    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884" event={"ID":"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8","Type":"ContainerStarted","Data":"c21f54f0389f5dd90f44f9a04655fcdedd7d8f195eb31ae6ccbfac7c7cab8a66"}
Oct 30 01:13:43.038291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.038251    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9" podUID="8dcc55f9-46b7-4500-8891-fd949f2a972d" containerName="main" containerID="cri-o://07a14540cf63922aa28c9ec8a27babd68d9859bd93d0cd338c7148f5a1fdba53" gracePeriod=45
Oct 30 01:13:43.125321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.125203    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9" podUID="8dcc55f9-46b7-4500-8891-fd949f2a972d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": dial tcp 10.133.0.107:8000: connect: connection refused"
Oct 30 01:13:43.187114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.187069    4576 generic.go:356] "Generic (PLEG): container finished" podID="8dcc55f9-46b7-4500-8891-fd949f2a972d" containerID="07a14540cf63922aa28c9ec8a27babd68d9859bd93d0cd338c7148f5a1fdba53" exitCode=0
Oct 30 01:13:43.187607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.187138    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9" event={"ID":"8dcc55f9-46b7-4500-8891-fd949f2a972d","Type":"ContainerDied","Data":"07a14540cf63922aa28c9ec8a27babd68d9859bd93d0cd338c7148f5a1fdba53"}
Oct 30 01:13:43.189069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.189040    4576 generic.go:356] "Generic (PLEG): container finished" podID="868ab4a6-9d55-4acf-8ef0-63c8a5c626f8" containerID="9a553e082107de2688e537bf04b862177cb0346e953d45b24c4c0b4899f3b8f2" exitCode=0
Oct 30 01:13:43.189191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.189123    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884" event={"ID":"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8","Type":"ContainerDied","Data":"9a553e082107de2688e537bf04b862177cb0346e953d45b24c4c0b4899f3b8f2"}
Oct 30 01:13:43.332036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.332009    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:13:43.369958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.369924    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dcc55f9-46b7-4500-8891-fd949f2a972d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcc55f9-46b7-4500-8891-fd949f2a972d-tls-certs\") pod \"8dcc55f9-46b7-4500-8891-fd949f2a972d\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") "
Oct 30 01:13:43.370137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.369983    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-home\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-home\") pod \"8dcc55f9-46b7-4500-8891-fd949f2a972d\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") "
Oct 30 01:13:43.370137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370062    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-model-cache\") pod \"8dcc55f9-46b7-4500-8891-fd949f2a972d\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") "
Oct 30 01:13:43.370137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370128    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dcc55f9-46b7-4500-8891-fd949f2a972d-kube-api-access-7hfsl\" (UniqueName: \"kubernetes.io/projected/8dcc55f9-46b7-4500-8891-fd949f2a972d-kube-api-access-7hfsl\") pod \"8dcc55f9-46b7-4500-8891-fd949f2a972d\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") "
Oct 30 01:13:43.370328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370163    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-tmp-dir\") pod \"8dcc55f9-46b7-4500-8891-fd949f2a972d\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") "
Oct 30 01:13:43.370328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370223    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-dshm\") pod \"8dcc55f9-46b7-4500-8891-fd949f2a972d\" (UID: \"8dcc55f9-46b7-4500-8891-fd949f2a972d\") "
Oct 30 01:13:43.370463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370312    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-home" pod "8dcc55f9-46b7-4500-8891-fd949f2a972d" (UID: "8dcc55f9-46b7-4500-8891-fd949f2a972d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:43.370519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370467    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-model-cache" pod "8dcc55f9-46b7-4500-8891-fd949f2a972d" (UID: "8dcc55f9-46b7-4500-8891-fd949f2a972d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:43.370519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370486    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-tmp-dir" pod "8dcc55f9-46b7-4500-8891-fd949f2a972d" (UID: "8dcc55f9-46b7-4500-8891-fd949f2a972d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:43.371014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370939    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:43.371014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370963    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:43.371014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.370979    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:43.372779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.372727    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dcc55f9-46b7-4500-8891-fd949f2a972d-tls-certs" pod "8dcc55f9-46b7-4500-8891-fd949f2a972d" (UID: "8dcc55f9-46b7-4500-8891-fd949f2a972d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:43.372893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.372845    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-dshm" pod "8dcc55f9-46b7-4500-8891-fd949f2a972d" (UID: "8dcc55f9-46b7-4500-8891-fd949f2a972d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:43.372936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.372905    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcc55f9-46b7-4500-8891-fd949f2a972d-kube-api-access-7hfsl" pod "8dcc55f9-46b7-4500-8891-fd949f2a972d" (UID: "8dcc55f9-46b7-4500-8891-fd949f2a972d"). InnerVolumeSpecName "kube-api-access-7hfsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:43.461565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.461535    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:43.472885 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.472852    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hfsl\" (UniqueName: \"kubernetes.io/projected/8dcc55f9-46b7-4500-8891-fd949f2a972d-kube-api-access-7hfsl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:43.472885 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.472884    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcc55f9-46b7-4500-8891-fd949f2a972d-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:43.473058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:43.472894    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcc55f9-46b7-4500-8891-fd949f2a972d-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:44.197722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.197689    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"
Oct 30 01:13:44.197722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.197712    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9" event={"ID":"8dcc55f9-46b7-4500-8891-fd949f2a972d","Type":"ContainerDied","Data":"f5eb35c219ceddd76dedede29b7429326831386022ca187eb930d583fb52b16c"}
Oct 30 01:13:44.198209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.197765    4576 scope.go:122] "RemoveContainer" containerID="07a14540cf63922aa28c9ec8a27babd68d9859bd93d0cd338c7148f5a1fdba53"
Oct 30 01:13:44.256888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.256860    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"]
Oct 30 01:13:44.271760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.271728    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-286d36/test-model-286d36-kserve-8584c47669-cdxd9"]
Oct 30 01:13:44.353521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.353494    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:44.385283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.385212    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-openshift-ca\") pod \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") "
Oct 30 01:13:44.385438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.385414    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-kube-api-access-wfcc6\" (UniqueName: \"kubernetes.io/projected/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-kube-api-access-wfcc6\") pod \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") "
Oct 30 01:13:44.385483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.385474    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-xks-ca\") pod \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\" (UID: \"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8\") "
Oct 30 01:13:44.385584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.385548    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-openshift-ca" pod "868ab4a6-9d55-4acf-8ef0-63c8a5c626f8" (UID: "868ab4a6-9d55-4acf-8ef0-63c8a5c626f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:44.385867 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.385851    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:44.387641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.387618    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-xks-ca" pod "868ab4a6-9d55-4acf-8ef0-63c8a5c626f8" (UID: "868ab4a6-9d55-4acf-8ef0-63c8a5c626f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:44.387760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.387690    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-kube-api-access-wfcc6" pod "868ab4a6-9d55-4acf-8ef0-63c8a5c626f8" (UID: "868ab4a6-9d55-4acf-8ef0-63c8a5c626f8"). InnerVolumeSpecName "kube-api-access-wfcc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:44.493657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.492435    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfcc6\" (UniqueName: \"kubernetes.io/projected/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-kube-api-access-wfcc6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:44.493657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.492484    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:44.597411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:44.597362    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" podUID="a475c6d4-042d-45cd-850e-49d0eda403c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:13:45.213272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:45.210194    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884" event={"ID":"868ab4a6-9d55-4acf-8ef0-63c8a5c626f8","Type":"ContainerDied","Data":"c21f54f0389f5dd90f44f9a04655fcdedd7d8f195eb31ae6ccbfac7c7cab8a66"}
Oct 30 01:13:45.213272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:45.210278    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c21f54f0389f5dd90f44f9a04655fcdedd7d8f195eb31ae6ccbfac7c7cab8a66"
Oct 30 01:13:45.213272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:45.210352    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"
Oct 30 01:13:45.395259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:45.394650    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:46.109770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:46.109730    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dcc55f9-46b7-4500-8891-fd949f2a972d" path="/var/lib/kubelet/pods/8dcc55f9-46b7-4500-8891-fd949f2a972d/volumes"
Oct 30 01:13:47.039507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:47.039460    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:48.383591 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:48.383557    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:13:48.559898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:48.559853    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:49.925532 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:49.925485    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:50.509622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:50.509580    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:50.602839 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:50.602804    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"]
Oct 30 01:13:50.650125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:50.650077    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:50.754363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:50.754321    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"]
Oct 30 01:13:50.758117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:50.758089    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d54608-flq7x"]
Oct 30 01:13:50.766514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:50.766439    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:50.917052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:50.917018    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:51.265147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:51.265115    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:52.117014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:52.116972    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eeafc9e-0d86-4c0f-ae49-fb8347223055" path="/var/lib/kubelet/pods/2eeafc9e-0d86-4c0f-ae49-fb8347223055/volumes"
Oct 30 01:13:52.121107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:52.121070    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"]
Oct 30 01:13:52.241967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:52.241927    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:52.350696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:52.350659    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:52.515078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:52.515042    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:53.260379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:53.260345    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:53.736338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:53.736303    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:53.856151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:53.856119    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"]
Oct 30 01:13:53.859498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:53.859472    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:53.859908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:53.859888    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-286d36-zv884"]
Oct 30 01:13:54.110425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.110347    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868ab4a6-9d55-4acf-8ef0-63c8a5c626f8" path="/var/lib/kubelet/pods/868ab4a6-9d55-4acf-8ef0-63c8a5c626f8/volumes"
Oct 30 01:13:54.314799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.314761    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:54.595785 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:13:54.595762    4576 logging.go:55] [core] [Channel #2498 SubChannel #2499]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.96:50052", ServerName: "10.133.0.96:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.96:50052: connect: connection refused"
Oct 30 01:13:54.844913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.844881    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:13:54.941770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.941738    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a475c6d4-042d-45cd-850e-49d0eda403c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a475c6d4-042d-45cd-850e-49d0eda403c5-plugins-config-volume\") pod \"a475c6d4-042d-45cd-850e-49d0eda403c5\" (UID: \"a475c6d4-042d-45cd-850e-49d0eda403c5\") "
Oct 30 01:13:54.941947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.941785    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a475c6d4-042d-45cd-850e-49d0eda403c5-kube-api-access-q42l4\" (UniqueName: \"kubernetes.io/projected/a475c6d4-042d-45cd-850e-49d0eda403c5-kube-api-access-q42l4\") pod \"a475c6d4-042d-45cd-850e-49d0eda403c5\" (UID: \"a475c6d4-042d-45cd-850e-49d0eda403c5\") "
Oct 30 01:13:54.942174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.942148    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a475c6d4-042d-45cd-850e-49d0eda403c5-plugins-config-volume" pod "a475c6d4-042d-45cd-850e-49d0eda403c5" (UID: "a475c6d4-042d-45cd-850e-49d0eda403c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:54.942330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.942313    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a475c6d4-042d-45cd-850e-49d0eda403c5-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:54.944138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.944113    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a475c6d4-042d-45cd-850e-49d0eda403c5-kube-api-access-q42l4" pod "a475c6d4-042d-45cd-850e-49d0eda403c5" (UID: "a475c6d4-042d-45cd-850e-49d0eda403c5"). InnerVolumeSpecName "kube-api-access-q42l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:54.984392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:54.984362    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:55.044393 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.044352    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q42l4\" (UniqueName: \"kubernetes.io/projected/a475c6d4-042d-45cd-850e-49d0eda403c5-kube-api-access-q42l4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:55.312354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.312321    4576 generic.go:356] "Generic (PLEG): container finished" podID="a475c6d4-042d-45cd-850e-49d0eda403c5" containerID="adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7" exitCode=0
Oct 30 01:13:55.312569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.312376    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"
Oct 30 01:13:55.312569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.312376    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" event={"ID":"a475c6d4-042d-45cd-850e-49d0eda403c5","Type":"ContainerDied","Data":"adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7"}
Oct 30 01:13:55.312569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.312435    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" event={"ID":"a475c6d4-042d-45cd-850e-49d0eda403c5","Type":"ContainerDied","Data":"6515a2e7799bf9b831288c6930a93cf7124ddeb9667fcbf54ac691cd84139e7a"}
Oct 30 01:13:55.312569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.312462    4576 scope.go:122] "RemoveContainer" containerID="adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7"
Oct 30 01:13:55.323830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.323806    4576 scope.go:122] "RemoveContainer" containerID="adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7"
Oct 30 01:13:55.324138 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:13:55.324116    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7\": container with ID starting with adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7 not found: ID does not exist" containerID="adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7"
Oct 30 01:13:55.324205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.324146    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7"} err="failed to get container status \"adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7\": rpc error: code = NotFound desc = could not find container \"adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7\": container with ID starting with adfcb5e1f1795b30b06e607ad2ffaa0120744940186b04693ff9bd412ddb59e7 not found: ID does not exist"
Oct 30 01:13:55.369251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.369203    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"]
Oct 30 01:13:55.382646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.382619    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv"]
Oct 30 01:13:55.595667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:55.595584    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d54608-59cb54d8dc-t77nv" podUID="a475c6d4-042d-45cd-850e-49d0eda403c5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.96:50052\" within 1s: context deadline exceeded"
Oct 30 01:13:56.108611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.108579    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a475c6d4-042d-45cd-850e-49d0eda403c5" path="/var/lib/kubelet/pods/a475c6d4-042d-45cd-850e-49d0eda403c5/volumes"
Oct 30 01:13:56.194550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.194513    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9" podUID="9a1c927d-c2cc-44ef-a2b9-323e0866a101" containerName="main" containerID="cri-o://4c84b082948a5f609ef5cafcb2ade85bdc60a20e9fe856c67918b780f3161d3f" gracePeriod=45
Oct 30 01:13:56.278548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.278514    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9" podUID="9a1c927d-c2cc-44ef-a2b9-323e0866a101" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": dial tcp 10.133.0.110:8000: connect: connection refused"
Oct 30 01:13:56.322723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.322686    4576 generic.go:356] "Generic (PLEG): container finished" podID="9a1c927d-c2cc-44ef-a2b9-323e0866a101" containerID="4c84b082948a5f609ef5cafcb2ade85bdc60a20e9fe856c67918b780f3161d3f" exitCode=0
Oct 30 01:13:56.322874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.322717    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9" event={"ID":"9a1c927d-c2cc-44ef-a2b9-323e0866a101","Type":"ContainerDied","Data":"4c84b082948a5f609ef5cafcb2ade85bdc60a20e9fe856c67918b780f3161d3f"}
Oct 30 01:13:56.473710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.473677    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:13:56.536391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.536353    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:56.562545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.562505    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-dshm\") pod \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") "
Oct 30 01:13:56.562767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.562560    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tmp-dir\") pod \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") "
Oct 30 01:13:56.562767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.562602    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1c927d-c2cc-44ef-a2b9-323e0866a101-kube-api-access-wg962\" (UniqueName: \"kubernetes.io/projected/9a1c927d-c2cc-44ef-a2b9-323e0866a101-kube-api-access-wg962\") pod \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") "
Oct 30 01:13:56.562767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.562713    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-model-cache\") pod \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") "
Oct 30 01:13:56.562923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.562771    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-home\") pod \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") "
Oct 30 01:13:56.562923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.562859    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tls-certs\") pod \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\" (UID: \"9a1c927d-c2cc-44ef-a2b9-323e0866a101\") "
Oct 30 01:13:56.563028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.562914    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tmp-dir" pod "9a1c927d-c2cc-44ef-a2b9-323e0866a101" (UID: "9a1c927d-c2cc-44ef-a2b9-323e0866a101"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:56.563416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.563129    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-home" pod "9a1c927d-c2cc-44ef-a2b9-323e0866a101" (UID: "9a1c927d-c2cc-44ef-a2b9-323e0866a101"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:56.563416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.563162    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-model-cache" pod "9a1c927d-c2cc-44ef-a2b9-323e0866a101" (UID: "9a1c927d-c2cc-44ef-a2b9-323e0866a101"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:56.563638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.563466    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:56.563638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.563504    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:56.563638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.563514    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:56.567016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.566929    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tls-certs" pod "9a1c927d-c2cc-44ef-a2b9-323e0866a101" (UID: "9a1c927d-c2cc-44ef-a2b9-323e0866a101"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:56.567171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.567037    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1c927d-c2cc-44ef-a2b9-323e0866a101-kube-api-access-wg962" pod "9a1c927d-c2cc-44ef-a2b9-323e0866a101" (UID: "9a1c927d-c2cc-44ef-a2b9-323e0866a101"). InnerVolumeSpecName "kube-api-access-wg962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:56.567280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.567199    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-dshm" pod "9a1c927d-c2cc-44ef-a2b9-323e0866a101" (UID: "9a1c927d-c2cc-44ef-a2b9-323e0866a101"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:56.666186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.665637    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c927d-c2cc-44ef-a2b9-323e0866a101-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:56.666186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.665675    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c927d-c2cc-44ef-a2b9-323e0866a101-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:56.666186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:56.665764    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg962\" (UniqueName: \"kubernetes.io/projected/9a1c927d-c2cc-44ef-a2b9-323e0866a101-kube-api-access-wg962\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:13:57.095739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.095699    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:57.121925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.121884    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv" podUID="222f8572-e0eb-4f90-a080-7d3d7ce1a047" containerName="payload-pre-processing" containerID="cri-o://c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db" gracePeriod=25
Oct 30 01:13:57.212344 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.212298    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:57.333310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.333276    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9" event={"ID":"9a1c927d-c2cc-44ef-a2b9-323e0866a101","Type":"ContainerDied","Data":"549441a8e232f8a47f3ee0c3e1f13859840232017734226c608f6876b6ec6f6e"}
Oct 30 01:13:57.333510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.333283    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"
Oct 30 01:13:57.333510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.333333    4576 scope.go:122] "RemoveContainer" containerID="4c84b082948a5f609ef5cafcb2ade85bdc60a20e9fe856c67918b780f3161d3f"
Oct 30 01:13:57.381464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.381432    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"]
Oct 30 01:13:57.381961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.381932    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy" containerID="cri-o://4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f" gracePeriod=30
Oct 30 01:13:57.429029 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.428997    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"]
Oct 30 01:13:57.441798 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.441769    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bc6092/test-model-bc6092-kserve-7d7b64f75f-nhcq9"]
Oct 30 01:13:57.536258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:57.527951    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:58.112502 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:58.112465    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1c927d-c2cc-44ef-a2b9-323e0866a101" path="/var/lib/kubelet/pods/9a1c927d-c2cc-44ef-a2b9-323e0866a101/volumes"
Oct 30 01:13:58.368419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:58.368316    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:59.831530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:59.831496    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:13:59.894770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:13:59.894730    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv" podUID="222f8572-e0eb-4f90-a080-7d3d7ce1a047" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:14:00.385380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:00.385346    4576 patch_prober.go:29] interesting pod/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:14:00.385563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:00.385398    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:14:00.390805 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:00.390775    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:00.503372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:00.503332    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:00.667004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:00.666918    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"]
Oct 30 01:14:00.667224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:00.667195    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy" containerID="cri-o://9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1" gracePeriod=30
Oct 30 01:14:00.717460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:00.717424    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:01.034996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.034957    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"]
Oct 30 01:14:01.092779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.092747    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:01.149138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.149105    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"]
Oct 30 01:14:01.150324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.150298    4576 cpu_manager.go:414] "containerMap: removing container" podUID="9a1c927d-c2cc-44ef-a2b9-323e0866a101" containerName="main"
Oct 30 01:14:01.150502 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.150468    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1c927d-c2cc-44ef-a2b9-323e0866a101" containerName="main"
Oct 30 01:14:01.150653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.150640    4576 cpu_manager.go:414] "containerMap: removing container" podUID="868ab4a6-9d55-4acf-8ef0-63c8a5c626f8" containerName="revoke-keys"
Oct 30 01:14:01.150734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.150724    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868ab4a6-9d55-4acf-8ef0-63c8a5c626f8" containerName="revoke-keys"
Oct 30 01:14:01.150816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.150807    4576 cpu_manager.go:414] "containerMap: removing container" podUID="a475c6d4-042d-45cd-850e-49d0eda403c5" containerName="payload-pre-processing"
Oct 30 01:14:01.150897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.150886    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a475c6d4-042d-45cd-850e-49d0eda403c5" containerName="payload-pre-processing"
Oct 30 01:14:01.151011 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.150999    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8dcc55f9-46b7-4500-8891-fd949f2a972d" containerName="main"
Oct 30 01:14:01.151115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.151102    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcc55f9-46b7-4500-8891-fd949f2a972d" containerName="main"
Oct 30 01:14:01.151410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.151392    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="868ab4a6-9d55-4acf-8ef0-63c8a5c626f8" containerName="revoke-keys"
Oct 30 01:14:01.151527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.151515    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="a475c6d4-042d-45cd-850e-49d0eda403c5" containerName="payload-pre-processing"
Oct 30 01:14:01.151616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.151606    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcc55f9-46b7-4500-8891-fd949f2a972d" containerName="main"
Oct 30 01:14:01.151706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.151697    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1c927d-c2cc-44ef-a2b9-323e0866a101" containerName="main"
Oct 30 01:14:01.161252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.161210    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.170360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.170324    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"]
Oct 30 01:14:01.198558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.198529    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:14:01.198902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.198883    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:14:01.359022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.358920    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflv7\" (UniqueName: \"kubernetes.io/projected/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-kube-api-access-rflv7\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.359298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.359071    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.359298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.359120    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.460350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.460313    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.460350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.460363    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.460632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.460504    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflv7\" (UniqueName: \"kubernetes.io/projected/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-kube-api-access-rflv7\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.461075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.461043    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.463109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.463080    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.483416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.483390    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflv7\" (UniqueName: \"kubernetes.io/projected/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-kube-api-access-rflv7\") pod \"maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.531950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.531921    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:01.820988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.820936    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:01.840400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.839143    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"]
Oct 30 01:14:01.847016 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:01.846970    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1aaeb84_f749_4cce_96a0_4dfd02b31db5.slice/crio-1ff30d7b8176147a828b1a151fb74522299fe49d6b0fca77bea8093dd2e2c80c WatchSource:0}: Error finding container 1ff30d7b8176147a828b1a151fb74522299fe49d6b0fca77bea8093dd2e2c80c: Status 404 returned error can't find the container with id 1ff30d7b8176147a828b1a151fb74522299fe49d6b0fca77bea8093dd2e2c80c
Oct 30 01:14:01.850780 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.850756    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"]
Oct 30 01:14:01.856801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.856783    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866012    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-token\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866206    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-podinfo\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866270    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdgp2\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-kube-api-access-fdgp2\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866303    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-certs\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866338    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e587ab-e7ae-4579-a632-ea1a760636bb-istiod-ca-cert\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866449    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-envoy\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866479    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-data\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866502    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-credential-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.866772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.866610    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.877743 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.877708    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-95fe33-openshift-default-dockercfg-t66db\""
Oct 30 01:14:01.882703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.882681    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"]
Oct 30 01:14:01.967667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.967624    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-data\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.967840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.967687    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-credential-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.967929 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.967839    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.967986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.967969    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-token\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.968045 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.968014    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-podinfo\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.968097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.968045    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdgp2\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-kube-api-access-fdgp2\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.968097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.968073    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-certs\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.968097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.968086    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-data\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.968270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.968110    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e587ab-e7ae-4579-a632-ea1a760636bb-istiod-ca-cert\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.968270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.968152    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-envoy\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.968577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.968555    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-credential-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.968694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.968624    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-socket\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.969163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.969097    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-certs\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.969411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.969212    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e587ab-e7ae-4579-a632-ea1a760636bb-istiod-ca-cert\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.971091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.971066    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-podinfo\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.971176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.971102    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-envoy\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.989565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.989540    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-token\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:01.989841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:01.989823    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdgp2\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-kube-api-access-fdgp2\") pod \"e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") " pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:02.184046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.184022    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:02.372617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.372521    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v" event={"ID":"f1aaeb84-f749-4cce-96a0-4dfd02b31db5","Type":"ContainerStarted","Data":"fabb659117c8c6e9ec2af4cca771906a7050684b51ae10ade5d4c71d3308019f"}
Oct 30 01:14:02.372617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.372589    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v" event={"ID":"f1aaeb84-f749-4cce-96a0-4dfd02b31db5","Type":"ContainerStarted","Data":"1ff30d7b8176147a828b1a151fb74522299fe49d6b0fca77bea8093dd2e2c80c"}
Oct 30 01:14:02.413913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.413866    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v" podStartSLOduration=1.4138525259999999 podStartE2EDuration="1.413852526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:02.413042162 +0000 UTC m=+2122.484365997" watchObservedRunningTime="2026-10-30 01:14:02.413852526 +0000 UTC m=+2122.485176360"
Oct 30 01:14:02.498753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.498723    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"]
Oct 30 01:14:02.498913 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:02.498888    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e587ab_e7ae_4579_a632_ea1a760636bb.slice/crio-bc87646c2263d3d6f543c9d9646e260257bf6a8fae18fc9b8e6abb6a0605c3c8 WatchSource:0}: Error finding container bc87646c2263d3d6f543c9d9646e260257bf6a8fae18fc9b8e6abb6a0605c3c8: Status 404 returned error can't find the container with id bc87646c2263d3d6f543c9d9646e260257bf6a8fae18fc9b8e6abb6a0605c3c8
Oct 30 01:14:02.502166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.502128    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:02.502320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.502200    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:02.502320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.502288    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:02.692769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.692738    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:14:02.779035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.778996    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-envoy\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.779251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779068    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-data\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.779251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779092    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-podinfo\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.779251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779148    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f069a5db-2229-42fd-9a0b-83f5e3365174-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f069a5db-2229-42fd-9a0b-83f5e3365174-istiod-ca-cert\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.779251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779213    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-certs\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.779477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779278    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-socket\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.779477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779394    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-token\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-token\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.779554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779474    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-kube-api-access-hxkvf\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-kube-api-access-hxkvf\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.779554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779512    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-credential-socket\") pod \"f069a5db-2229-42fd-9a0b-83f5e3365174\" (UID: \"f069a5db-2229-42fd-9a0b-83f5e3365174\") "
Oct 30 01:14:02.780500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779396    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-data" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:02.780630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.779761    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f069a5db-2229-42fd-9a0b-83f5e3365174-istiod-ca-cert" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:02.780630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.780340    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-certs" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:02.780630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.780449    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-socket" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:02.780630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.780581    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-credential-socket" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:02.782093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.782067    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-envoy" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:02.782438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.782409    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-podinfo" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:14:02.783788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.783765    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-token" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:02.784869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.784848    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-kube-api-access-hxkvf" pod "f069a5db-2229-42fd-9a0b-83f5e3365174" (UID: "f069a5db-2229-42fd-9a0b-83f5e3365174"). InnerVolumeSpecName "kube-api-access-hxkvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.881879    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.881926    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.881942    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.881956    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f069a5db-2229-42fd-9a0b-83f5e3365174-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.881979    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.881993    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.882005    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.882018    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxkvf\" (UniqueName: \"kubernetes.io/projected/f069a5db-2229-42fd-9a0b-83f5e3365174-kube-api-access-hxkvf\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.884398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:02.882037    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f069a5db-2229-42fd-9a0b-83f5e3365174-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:03.390265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.390194    4576 generic.go:356] "Generic (PLEG): container finished" podID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerID="4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f" exitCode=0
Oct 30 01:14:03.390927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.390901    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" event={"ID":"f069a5db-2229-42fd-9a0b-83f5e3365174","Type":"ContainerDied","Data":"4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f"}
Oct 30 01:14:03.391104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.391088    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b" event={"ID":"f069a5db-2229-42fd-9a0b-83f5e3365174","Type":"ContainerDied","Data":"5c5fdede67fd33383ba8fbcdd81d8c6117c8963843f7ba1b3bf5cb24d31f8566"}
Oct 30 01:14:03.391251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.391216    4576 scope.go:122] "RemoveContainer" containerID="4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f"
Oct 30 01:14:03.391535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.391517    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"
Oct 30 01:14:03.396910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.396880    4576 generic.go:356] "Generic (PLEG): container finished" podID="f1aaeb84-f749-4cce-96a0-4dfd02b31db5" containerID="fabb659117c8c6e9ec2af4cca771906a7050684b51ae10ade5d4c71d3308019f" exitCode=0
Oct 30 01:14:03.397115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.396964    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v" event={"ID":"f1aaeb84-f749-4cce-96a0-4dfd02b31db5","Type":"ContainerDied","Data":"fabb659117c8c6e9ec2af4cca771906a7050684b51ae10ade5d4c71d3308019f"}
Oct 30 01:14:03.399260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.399218    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" event={"ID":"90e587ab-e7ae-4579-a632-ea1a760636bb","Type":"ContainerStarted","Data":"7a222c104121cb157359eaff6fcce0606b289b97b14ebb355fa60159d9ecae5a"}
Oct 30 01:14:03.399365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.399288    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" event={"ID":"90e587ab-e7ae-4579-a632-ea1a760636bb","Type":"ContainerStarted","Data":"bc87646c2263d3d6f543c9d9646e260257bf6a8fae18fc9b8e6abb6a0605c3c8"}
Oct 30 01:14:03.408284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.408199    4576 scope.go:122] "RemoveContainer" containerID="c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8"
Oct 30 01:14:03.418568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.418548    4576 scope.go:122] "RemoveContainer" containerID="4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f"
Oct 30 01:14:03.418857 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:03.418836    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f\": container with ID starting with 4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f not found: ID does not exist" containerID="4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f"
Oct 30 01:14:03.418930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.418866    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f"} err="failed to get container status \"4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f\": rpc error: code = NotFound desc = could not find container \"4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f\": container with ID starting with 4ad5c5ada5e94a3482f975d014bee1c88e0714dac6bae773b8886b3e022d4d7f not found: ID does not exist"
Oct 30 01:14:03.418930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.418884    4576 scope.go:122] "RemoveContainer" containerID="c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8"
Oct 30 01:14:03.419140 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:03.419124    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8\": container with ID starting with c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8 not found: ID does not exist" containerID="c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8"
Oct 30 01:14:03.419188 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.419146    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8"} err="failed to get container status \"c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8\": rpc error: code = NotFound desc = could not find container \"c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8\": container with ID starting with c901a3651402b5cfbe44beff02cdb2eb7e7baf6d2ac3df02692354b22e1f45e8 not found: ID does not exist"
Oct 30 01:14:03.496813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.496777    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"]
Oct 30 01:14:03.502531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.502505    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d54608-openshift-default-868c987dbc-4jg4b"]
Oct 30 01:14:03.509317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.509284    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:03.549511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:03.549425    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" podStartSLOduration=2.549394333 podStartE2EDuration="2.549394333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:03.53852945 +0000 UTC m=+2123.609853284" watchObservedRunningTime="2026-10-30 01:14:03.549394333 +0000 UTC m=+2123.620718169"
Oct 30 01:14:04.109973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.109940    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" path="/var/lib/kubelet/pods/f069a5db-2229-42fd-9a0b-83f5e3365174/volumes"
Oct 30 01:14:04.192000 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.191909    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:04.192954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.192914    4576 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 01:14:04.193098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.192973    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 01:14:04.641961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.641933    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:04.715401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.715369    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-openshift-ca\") pod \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") "
Oct 30 01:14:04.715711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.715690    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-kube-api-access-rflv7\" (UniqueName: \"kubernetes.io/projected/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-kube-api-access-rflv7\") pod \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") "
Oct 30 01:14:04.715928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.715910    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-xks-ca\") pod \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\" (UID: \"f1aaeb84-f749-4cce-96a0-4dfd02b31db5\") "
Oct 30 01:14:04.717218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.717186    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-openshift-ca" pod "f1aaeb84-f749-4cce-96a0-4dfd02b31db5" (UID: "f1aaeb84-f749-4cce-96a0-4dfd02b31db5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:04.720022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.719997    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-xks-ca" pod "f1aaeb84-f749-4cce-96a0-4dfd02b31db5" (UID: "f1aaeb84-f749-4cce-96a0-4dfd02b31db5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:04.721354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.721332    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-kube-api-access-rflv7" pod "f1aaeb84-f749-4cce-96a0-4dfd02b31db5" (UID: "f1aaeb84-f749-4cce-96a0-4dfd02b31db5"). InnerVolumeSpecName "kube-api-access-rflv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:04.817923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.817887    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:04.818142 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.818124    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflv7\" (UniqueName: \"kubernetes.io/projected/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-kube-api-access-rflv7\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:04.818260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:04.818224    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1aaeb84-f749-4cce-96a0-4dfd02b31db5-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:05.030352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.030268    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:05.197297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.192634    4576 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 01:14:05.197297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.192699    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 01:14:05.416005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.415908    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v" event={"ID":"f1aaeb84-f749-4cce-96a0-4dfd02b31db5","Type":"ContainerDied","Data":"1ff30d7b8176147a828b1a151fb74522299fe49d6b0fca77bea8093dd2e2c80c"}
Oct 30 01:14:05.416005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.415977    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ff30d7b8176147a828b1a151fb74522299fe49d6b0fca77bea8093dd2e2c80c"
Oct 30 01:14:05.416005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.415926    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"
Oct 30 01:14:05.626458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.626419    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk" podUID="0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" containerName="main" containerID="cri-o://f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0" gracePeriod=45
Oct 30 01:14:05.725842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.725811    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"]
Oct 30 01:14:05.726507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726481    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f1aaeb84-f749-4cce-96a0-4dfd02b31db5" containerName="revoke-keys"
Oct 30 01:14:05.726507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726504    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1aaeb84-f749-4cce-96a0-4dfd02b31db5" containerName="revoke-keys"
Oct 30 01:14:05.726691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726521    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy"
Oct 30 01:14:05.726691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726530    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy"
Oct 30 01:14:05.726691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726549    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy"
Oct 30 01:14:05.726691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726558    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy"
Oct 30 01:14:05.726831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726703    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1aaeb84-f749-4cce-96a0-4dfd02b31db5" containerName="revoke-keys"
Oct 30 01:14:05.726831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726711    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy"
Oct 30 01:14:05.726831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.726721    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f069a5db-2229-42fd-9a0b-83f5e3365174" containerName="istio-proxy"
Oct 30 01:14:05.730810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.730791    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.749131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.749081    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-43d152-openshift-default-dockercfg-5w798\""
Oct 30 01:14:05.753482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.753454    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"]
Oct 30 01:14:05.830915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.830883    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istiod-ca-cert\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.831074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.830950    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-envoy\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.831074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.830979    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.831074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.831013    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-data\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.831074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.831038    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-podinfo\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.831227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.831082    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklhg\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-kube-api-access-bklhg\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.831227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.831100    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-credential-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.831227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.831116    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-certs\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.831227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.831164    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-token\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934343    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-token\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934491    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istiod-ca-cert\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934599    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-envoy\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934643    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934694    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-data\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934743    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-podinfo\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934828    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklhg\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-kube-api-access-bklhg\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934860    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-credential-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.934946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.934888    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-certs\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.935550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.935390    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-certs\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.936483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.936454    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-credential-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.936945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.936924    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-data\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.937037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.937018    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-socket\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.937391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.937371    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istiod-ca-cert\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.939267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.939153    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-envoy\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.939680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.939659    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-podinfo\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.960341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.960268    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklhg\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-kube-api-access-bklhg\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:05.961357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:05.961332    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-token\") pod \"e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") " pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:06.029574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.029540    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:14:06.035718 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.035692    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-home\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-home\") pod \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") "
Oct 30 01:14:06.035853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.035770    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-dshm\") pod \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") "
Oct 30 01:14:06.035853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.035797    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tmp-dir\") pod \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") "
Oct 30 01:14:06.035853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.035839    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-kube-api-access-7mmv4\" (UniqueName: \"kubernetes.io/projected/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-kube-api-access-7mmv4\") pod \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") "
Oct 30 01:14:06.035946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.035863    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-model-cache\") pod \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") "
Oct 30 01:14:06.035946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.035923    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tls-certs\") pod \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\" (UID: \"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1\") "
Oct 30 01:14:06.036431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.036406    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tmp-dir" pod "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" (UID: "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.036646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.036621    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-model-cache" pod "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" (UID: "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.036739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.036720    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-home" pod "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" (UID: "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.039014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.038989    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-kube-api-access-7mmv4" pod "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" (UID: "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1"). InnerVolumeSpecName "kube-api-access-7mmv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:06.039108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.039065    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-dshm" pod "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" (UID: "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.039108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.039103    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tls-certs" pod "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" (UID: "0ab39d4c-b152-4fe4-93f1-4411e5fdddd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:06.057013 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.056989    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:14:06.116524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.116494    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:06.137711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137684    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-kube-api-access-gkhbz\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-kube-api-access-gkhbz\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.137837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137726    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d55436d-2557-4b59-8409-6739cd1b4551-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d55436d-2557-4b59-8409-6739cd1b4551-istiod-ca-cert\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.137837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137760    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-envoy\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.137837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137802    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-data\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.137837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137828    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-certs\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.138061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137858    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-socket\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.138061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137925    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-credential-socket\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.138061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137951    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d55436d-2557-4b59-8409-6739cd1b4551-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d55436d-2557-4b59-8409-6739cd1b4551-istio-podinfo\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.138061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.137971    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-istio-token\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-istio-token\") pod \"9d55436d-2557-4b59-8409-6739cd1b4551\" (UID: \"9d55436d-2557-4b59-8409-6739cd1b4551\") "
Oct 30 01:14:06.138482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.138415    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-socket" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.138579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.138506    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-data" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.138646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.138622    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-certs" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.138973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.138950    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-credential-socket" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.139277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.139256    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.139373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.139325    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.139373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.139345    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmv4\" (UniqueName: \"kubernetes.io/projected/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-kube-api-access-7mmv4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.139373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.139362    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.139568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.139376    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.139568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.139389    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.139568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.139401    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.139568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.139434    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d55436d-2557-4b59-8409-6739cd1b4551-istiod-ca-cert" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:06.153455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.153417    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-envoy" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:06.153455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.153422    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-istio-token" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:06.160736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.160695    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d55436d-2557-4b59-8409-6739cd1b4551-istio-podinfo" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:14:06.166442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.166413    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-kube-api-access-gkhbz" pod "9d55436d-2557-4b59-8409-6739cd1b4551" (UID: "9d55436d-2557-4b59-8409-6739cd1b4551"). InnerVolumeSpecName "kube-api-access-gkhbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:06.192260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.185651    4576 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 01:14:06.192260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.185710    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 01:14:06.243360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.243330    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkhbz\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-kube-api-access-gkhbz\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.243360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.243359    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d55436d-2557-4b59-8409-6739cd1b4551-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.243614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.243370    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.243614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.243378    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.243614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.243386    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.243614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.243395    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d55436d-2557-4b59-8409-6739cd1b4551-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.243614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.243403    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d55436d-2557-4b59-8409-6739cd1b4551-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.243614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.243410    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d55436d-2557-4b59-8409-6739cd1b4551-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:06.425083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.425048    4576 generic.go:356] "Generic (PLEG): container finished" podID="9d55436d-2557-4b59-8409-6739cd1b4551" containerID="9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1" exitCode=0
Oct 30 01:14:06.425308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.425287    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" event={"ID":"9d55436d-2557-4b59-8409-6739cd1b4551","Type":"ContainerDied","Data":"9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1"}
Oct 30 01:14:06.425427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.425332    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7" event={"ID":"9d55436d-2557-4b59-8409-6739cd1b4551","Type":"ContainerDied","Data":"742616ebcd4d4b72026b21a5452a72512fbeb192eda871678a072c193b51c2ae"}
Oct 30 01:14:06.425427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.425354    4576 scope.go:122] "RemoveContainer" containerID="9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1"
Oct 30 01:14:06.425512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.425456    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"
Oct 30 01:14:06.427189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.427168    4576 generic.go:356] "Generic (PLEG): container finished" podID="0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" containerID="f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0" exitCode=0
Oct 30 01:14:06.427318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.427219    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"
Oct 30 01:14:06.427391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.427281    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk" event={"ID":"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1","Type":"ContainerDied","Data":"f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0"}
Oct 30 01:14:06.427459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.427410    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk" event={"ID":"0ab39d4c-b152-4fe4-93f1-4411e5fdddd1","Type":"ContainerDied","Data":"8001f35948bc2ebaddbc3171399c2fab2dce0e6fee0e0db45b0e04c47f9b9a5d"}
Oct 30 01:14:06.444816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.444789    4576 scope.go:122] "RemoveContainer" containerID="9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1"
Oct 30 01:14:06.445176 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:06.445152    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1\": container with ID starting with 9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1 not found: ID does not exist" containerID="9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1"
Oct 30 01:14:06.445295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.445194    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1"} err="failed to get container status \"9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1\": rpc error: code = NotFound desc = could not find container \"9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1\": container with ID starting with 9b13c21730ce045b15ff2732d6cdc3debeec8139f3dceff6237b3f4b889fa7b1 not found: ID does not exist"
Oct 30 01:14:06.445295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.445220    4576 scope.go:122] "RemoveContainer" containerID="f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0"
Oct 30 01:14:06.456428 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.456407    4576 scope.go:122] "RemoveContainer" containerID="f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0"
Oct 30 01:14:06.456705 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:06.456684    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0\": container with ID starting with f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0 not found: ID does not exist" containerID="f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0"
Oct 30 01:14:06.456773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.456714    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0"} err="failed to get container status \"f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0\": rpc error: code = NotFound desc = could not find container \"f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0\": container with ID starting with f267e9731a650cc5c85a69cdfb1120f57586c10ed0ffa000e34c2b5bd5fe1ab0 not found: ID does not exist"
Oct 30 01:14:06.497801 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:06.497763    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f1033e_bfd7_49e9_9dcd_bb82d32dd11c.slice/crio-e32b0c93d20a8299d726b87be81ab7f77368d47cedb92b0b5eb9ef193fdef045 WatchSource:0}: Error finding container e32b0c93d20a8299d726b87be81ab7f77368d47cedb92b0b5eb9ef193fdef045: Status 404 returned error can't find the container with id e32b0c93d20a8299d726b87be81ab7f77368d47cedb92b0b5eb9ef193fdef045
Oct 30 01:14:06.501901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.500611    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"]
Oct 30 01:14:06.505165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.505140    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"]
Oct 30 01:14:06.505350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.505326    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:06.505409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.505384    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:06.505443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.505422    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:06.508791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.508771    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-286d36-openshift-default-869bc8c77-szwm7"]
Oct 30 01:14:06.531472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.531444    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"]
Oct 30 01:14:06.534756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.534732    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/auth-test-model-42d56b-kserve-75c689f87c-s9xwk"]
Oct 30 01:14:06.547940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:06.547916    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:07.192265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:07.188700    4576 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 01:14:07.192265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:07.188762    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 01:14:07.437622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:07.437585    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" event={"ID":"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c","Type":"ContainerStarted","Data":"70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9"}
Oct 30 01:14:07.437804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:07.437662    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" event={"ID":"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c","Type":"ContainerStarted","Data":"e32b0c93d20a8299d726b87be81ab7f77368d47cedb92b0b5eb9ef193fdef045"}
Oct 30 01:14:07.490315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:07.486476    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" podStartSLOduration=2.486453595 podStartE2EDuration="2.486453595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:07.484901883 +0000 UTC m=+2127.556225715" watchObservedRunningTime="2026-10-30 01:14:07.486453595 +0000 UTC m=+2127.557777432"
Oct 30 01:14:08.122256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:08.120327    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 01:14:08.122256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:08.120391    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 01:14:08.128442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:08.128406    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" path="/var/lib/kubelet/pods/0ab39d4c-b152-4fe4-93f1-4411e5fdddd1/volumes"
Oct 30 01:14:08.129035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:08.129020    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" path="/var/lib/kubelet/pods/9d55436d-2557-4b59-8409-6739cd1b4551/volumes"
Oct 30 01:14:08.129461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:08.129447    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:08.186269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:08.185722    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:08.186269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:08.186065    4576 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 01:14:08.186269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:08.186112    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 01:14:09.117521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:09.117489    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 01:14:09.117918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:09.117547    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 01:14:09.290266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:09.286201    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:09.290266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:09.287519    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:09.297491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:09.297463    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:14:09.692860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:09.692826    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:09.895248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:09.895194    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv" podUID="222f8572-e0eb-4f90-a080-7d3d7ce1a047" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:14:10.116875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:10.116795    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 01:14:10.116875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:10.116852    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 01:14:11.117680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:11.117645    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 01:14:11.118117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:11.117711    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 01:14:11.252501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:11.252458    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:11.599296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:11.599259    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"]
Oct 30 01:14:11.635835 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:11.635803    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"]
Oct 30 01:14:11.794968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:11.793118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:11.905586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:11.905363    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:12.117569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:12.117536    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 01:14:12.117768 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:12.117594    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 01:14:12.990628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:12.990596    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:13.277843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:13.277752    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:13.278891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:13.278865    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:13.280035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:13.280013    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:14:14.559082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:14.559052    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:16.047514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.047478    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:16.062090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.062051    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm" podUID="9dc98774-772b-4f71-99df-48d0b53a35b4" containerName="main" containerID="cri-o://232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420" gracePeriod=45
Oct 30 01:14:16.324210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.324183    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:14:16.367487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.367441    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-home\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-home\") pod \"9dc98774-772b-4f71-99df-48d0b53a35b4\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") "
Oct 30 01:14:16.367736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.367706    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-home" pod "9dc98774-772b-4f71-99df-48d0b53a35b4" (UID: "9dc98774-772b-4f71-99df-48d0b53a35b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:16.367862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.367838    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc98774-772b-4f71-99df-48d0b53a35b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc98774-772b-4f71-99df-48d0b53a35b4-tls-certs\") pod \"9dc98774-772b-4f71-99df-48d0b53a35b4\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") "
Oct 30 01:14:16.367994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.367946    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc98774-772b-4f71-99df-48d0b53a35b4-kube-api-access-dtfrr\" (UniqueName: \"kubernetes.io/projected/9dc98774-772b-4f71-99df-48d0b53a35b4-kube-api-access-dtfrr\") pod \"9dc98774-772b-4f71-99df-48d0b53a35b4\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") "
Oct 30 01:14:16.367994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.367974    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-model-cache\") pod \"9dc98774-772b-4f71-99df-48d0b53a35b4\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") "
Oct 30 01:14:16.368119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.368038    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-tmp-dir\") pod \"9dc98774-772b-4f71-99df-48d0b53a35b4\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") "
Oct 30 01:14:16.368119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.368089    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-dshm\") pod \"9dc98774-772b-4f71-99df-48d0b53a35b4\" (UID: \"9dc98774-772b-4f71-99df-48d0b53a35b4\") "
Oct 30 01:14:16.368587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.368495    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-model-cache" pod "9dc98774-772b-4f71-99df-48d0b53a35b4" (UID: "9dc98774-772b-4f71-99df-48d0b53a35b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:16.368673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.368628    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-tmp-dir" pod "9dc98774-772b-4f71-99df-48d0b53a35b4" (UID: "9dc98774-772b-4f71-99df-48d0b53a35b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:16.369090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.369066    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:16.369193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.369095    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:16.369193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.369111    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:16.370888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.370867    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc98774-772b-4f71-99df-48d0b53a35b4-tls-certs" pod "9dc98774-772b-4f71-99df-48d0b53a35b4" (UID: "9dc98774-772b-4f71-99df-48d0b53a35b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:16.370888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.370878    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc98774-772b-4f71-99df-48d0b53a35b4-kube-api-access-dtfrr" pod "9dc98774-772b-4f71-99df-48d0b53a35b4" (UID: "9dc98774-772b-4f71-99df-48d0b53a35b4"). InnerVolumeSpecName "kube-api-access-dtfrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:16.371023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.370961    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-dshm" pod "9dc98774-772b-4f71-99df-48d0b53a35b4" (UID: "9dc98774-772b-4f71-99df-48d0b53a35b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:16.470367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.470331    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc98774-772b-4f71-99df-48d0b53a35b4-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:16.470367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.470363    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfrr\" (UniqueName: \"kubernetes.io/projected/9dc98774-772b-4f71-99df-48d0b53a35b4-kube-api-access-dtfrr\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:16.470367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.470374    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc98774-772b-4f71-99df-48d0b53a35b4-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:16.528692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.528659    4576 generic.go:356] "Generic (PLEG): container finished" podID="9dc98774-772b-4f71-99df-48d0b53a35b4" containerID="232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420" exitCode=0
Oct 30 01:14:16.528882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.528714    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"
Oct 30 01:14:16.528882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.528746    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm" event={"ID":"9dc98774-772b-4f71-99df-48d0b53a35b4","Type":"ContainerDied","Data":"232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420"}
Oct 30 01:14:16.528882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.528792    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm" event={"ID":"9dc98774-772b-4f71-99df-48d0b53a35b4","Type":"ContainerDied","Data":"dca9e552153f2bba8d00f5d9a7ee0b8268f4a679cfc80bc7f3916fe69196b4f7"}
Oct 30 01:14:16.528882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.528809    4576 scope.go:122] "RemoveContainer" containerID="232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420"
Oct 30 01:14:16.539754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.539729    4576 scope.go:122] "RemoveContainer" containerID="232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420"
Oct 30 01:14:16.540048 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:16.540029    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420\": container with ID starting with 232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420 not found: ID does not exist" containerID="232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420"
Oct 30 01:14:16.540101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.540056    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420"} err="failed to get container status \"232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420\": rpc error: code = NotFound desc = could not find container \"232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420\": container with ID starting with 232df80a8189e14b45d713d6e3cc0c711cd0db0b167a5d68e5defb2108d64420 not found: ID does not exist"
Oct 30 01:14:16.585648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.585557    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"]
Oct 30 01:14:16.590900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.590874    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/auth-test-model-d0ae0d-kserve-5847cc7557-z72lm"]
Oct 30 01:14:16.600291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.600261    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9" podUID="2c1f64ae-fe2e-4c69-8e98-aab96c76b832" containerName="payload-processing" containerID="cri-o://79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a" gracePeriod=25
Oct 30 01:14:16.636450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:16.636417    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm" podUID="cbc93b7e-19c5-4609-b9b7-4515f213f848" containerName="payload-pre-processing" containerID="cri-o://e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522" gracePeriod=25
Oct 30 01:14:17.404428 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.404404    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:14:17.482139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.482098    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222f8572-e0eb-4f90-a080-7d3d7ce1a047-kube-api-access-9zsp8\" (UniqueName: \"kubernetes.io/projected/222f8572-e0eb-4f90-a080-7d3d7ce1a047-kube-api-access-9zsp8\") pod \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\" (UID: \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\") "
Oct 30 01:14:17.482336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.482219    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222f8572-e0eb-4f90-a080-7d3d7ce1a047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222f8572-e0eb-4f90-a080-7d3d7ce1a047-plugins-config-volume\") pod \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\" (UID: \"222f8572-e0eb-4f90-a080-7d3d7ce1a047\") "
Oct 30 01:14:17.482716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.482689    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222f8572-e0eb-4f90-a080-7d3d7ce1a047-plugins-config-volume" pod "222f8572-e0eb-4f90-a080-7d3d7ce1a047" (UID: "222f8572-e0eb-4f90-a080-7d3d7ce1a047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:17.482969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.482947    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222f8572-e0eb-4f90-a080-7d3d7ce1a047-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:17.484654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.484626    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222f8572-e0eb-4f90-a080-7d3d7ce1a047-kube-api-access-9zsp8" pod "222f8572-e0eb-4f90-a080-7d3d7ce1a047" (UID: "222f8572-e0eb-4f90-a080-7d3d7ce1a047"). InnerVolumeSpecName "kube-api-access-9zsp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:17.537952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.537911    4576 generic.go:356] "Generic (PLEG): container finished" podID="222f8572-e0eb-4f90-a080-7d3d7ce1a047" containerID="c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db" exitCode=0
Oct 30 01:14:17.538154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.537956    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"
Oct 30 01:14:17.538154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.538000    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv" event={"ID":"222f8572-e0eb-4f90-a080-7d3d7ce1a047","Type":"ContainerDied","Data":"c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db"}
Oct 30 01:14:17.538154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.538060    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv" event={"ID":"222f8572-e0eb-4f90-a080-7d3d7ce1a047","Type":"ContainerDied","Data":"3eeb3c806dc318e986dee8685fa4f4002deb9554d6de577f710158599d34be61"}
Oct 30 01:14:17.538154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.538090    4576 scope.go:122] "RemoveContainer" containerID="c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db"
Oct 30 01:14:17.551431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.551408    4576 scope.go:122] "RemoveContainer" containerID="c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db"
Oct 30 01:14:17.551732 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:17.551701    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db\": container with ID starting with c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db not found: ID does not exist" containerID="c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db"
Oct 30 01:14:17.551834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.551740    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db"} err="failed to get container status \"c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db\": rpc error: code = NotFound desc = could not find container \"c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db\": container with ID starting with c16ef64306a2a8897e4788db4a7a881eea57fa0405c4ff7674b9844ca04a76db not found: ID does not exist"
Oct 30 01:14:17.583831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.583803    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zsp8\" (UniqueName: \"kubernetes.io/projected/222f8572-e0eb-4f90-a080-7d3d7ce1a047-kube-api-access-9zsp8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:17.595270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.595245    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"]
Oct 30 01:14:17.604154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.604131    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-286d36-598cb6b6c5-rn6zv"]
Oct 30 01:14:17.658907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.658832    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:17.706392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.706352    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm" podUID="cbc93b7e-19c5-4609-b9b7-4515f213f848" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:14:17.710499 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.710468    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9" podUID="2c1f64ae-fe2e-4c69-8e98-aab96c76b832" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:14:17.710999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.710980    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"]
Oct 30 01:14:17.711582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711564    4576 cpu_manager.go:414] "containerMap: removing container" podUID="0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" containerName="main"
Oct 30 01:14:17.711582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711581    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" containerName="main"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711609    4576 cpu_manager.go:414] "containerMap: removing container" podUID="222f8572-e0eb-4f90-a080-7d3d7ce1a047" containerName="payload-pre-processing"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711614    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222f8572-e0eb-4f90-a080-7d3d7ce1a047" containerName="payload-pre-processing"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711627    4576 cpu_manager.go:414] "containerMap: removing container" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711632    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711649    4576 cpu_manager.go:414] "containerMap: removing container" podUID="9dc98774-772b-4f71-99df-48d0b53a35b4" containerName="main"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711654    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc98774-772b-4f71-99df-48d0b53a35b4" containerName="main"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711745    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc98774-772b-4f71-99df-48d0b53a35b4" containerName="main"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711756    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab39d4c-b152-4fe4-93f1-4411e5fdddd1" containerName="main"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711766    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="222f8572-e0eb-4f90-a080-7d3d7ce1a047" containerName="payload-pre-processing"
Oct 30 01:14:17.711769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.711774    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d55436d-2557-4b59-8409-6739cd1b4551" containerName="istio-proxy"
Oct 30 01:14:17.716533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.716515    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.731828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.731787    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"]
Oct 30 01:14:17.732927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.732766    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-b05110\"/\"openshift-service-ca.crt\""
Oct 30 01:14:17.733094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.733015    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-b05110\"/\"default-dockercfg-khgbq\""
Oct 30 01:14:17.733094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.733084    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-b05110\"/\"auth-test-model-b05110-kserve-self-signed-certs\""
Oct 30 01:14:17.733273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.733134    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-b05110\"/\"kube-root-ca.crt\""
Oct 30 01:14:17.775149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.775101    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:17.785735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.785710    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-home\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.785913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.785756    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-dshm\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.785913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.785891    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d229ebd4-d9ec-4495-bb5e-476328755569-tls-certs\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.786094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.786079    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-model-cache\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.786137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.786111    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-775v9\" (UniqueName: \"kubernetes.io/projected/d229ebd4-d9ec-4495-bb5e-476328755569-kube-api-access-775v9\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.786179 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.786141    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-tmp-dir\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.891374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.888008    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-model-cache\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.891374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.888071    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-775v9\" (UniqueName: \"kubernetes.io/projected/d229ebd4-d9ec-4495-bb5e-476328755569-kube-api-access-775v9\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.891374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.888119    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-tmp-dir\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.891374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.888187    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-home\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.891374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.888256    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-dshm\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.891374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.888297    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d229ebd4-d9ec-4495-bb5e-476328755569-tls-certs\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.895263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.892589    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-model-cache\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.895263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.893162    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-home\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.895263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.893680    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-tmp-dir\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.896299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.896140    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-dshm\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.898930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.898909    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d229ebd4-d9ec-4495-bb5e-476328755569-tls-certs\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:17.914183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:17.914120    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-775v9\" (UniqueName: \"kubernetes.io/projected/d229ebd4-d9ec-4495-bb5e-476328755569-kube-api-access-775v9\") pod \"auth-test-model-b05110-kserve-7bf77f56b5-clbps\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") " pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:18.034545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:18.034222    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:18.127419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:18.127357    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222f8572-e0eb-4f90-a080-7d3d7ce1a047" path="/var/lib/kubelet/pods/222f8572-e0eb-4f90-a080-7d3d7ce1a047/volumes"
Oct 30 01:14:18.128325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:18.128292    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc98774-772b-4f71-99df-48d0b53a35b4" path="/var/lib/kubelet/pods/9dc98774-772b-4f71-99df-48d0b53a35b4/volumes"
Oct 30 01:14:18.341471 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:18.341442    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"]
Oct 30 01:14:18.342576 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:18.342546    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd229ebd4_d9ec_4495_bb5e_476328755569.slice/crio-ca8cb323f33629292f961422aa892d0cae1ce03cc8ccd959b48a5985d8b8b458 WatchSource:0}: Error finding container ca8cb323f33629292f961422aa892d0cae1ce03cc8ccd959b48a5985d8b8b458: Status 404 returned error can't find the container with id ca8cb323f33629292f961422aa892d0cae1ce03cc8ccd959b48a5985d8b8b458
Oct 30 01:14:18.386041 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:18.386012    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:14:18.548616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:18.548581    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps" event={"ID":"d229ebd4-d9ec-4495-bb5e-476328755569","Type":"ContainerStarted","Data":"5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a"}
Oct 30 01:14:18.548989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:18.548625    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps" event={"ID":"d229ebd4-d9ec-4495-bb5e-476328755569","Type":"ContainerStarted","Data":"ca8cb323f33629292f961422aa892d0cae1ce03cc8ccd959b48a5985d8b8b458"}
Oct 30 01:14:18.590663 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:18.590568    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps" podStartSLOduration=1.5905542160000001 podStartE2EDuration="1.590554216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:18.589515768 +0000 UTC m=+2138.660839606" watchObservedRunningTime="2026-10-30 01:14:18.590554216 +0000 UTC m=+2138.661878103"
Oct 30 01:14:19.035012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:19.034979    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:19.376435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:19.376347    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:19.915851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:19.915816    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:20.291285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:20.287927    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:14:20.339823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:20.339793    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:20.823368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:20.823327    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:20.957361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:20.957324    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:21.075950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:21.075851    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:23.396680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:23.396631    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:25.149807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:25.149767    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:25.279875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:25.279838    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:25.515650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:25.515613    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:25.635267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:25.635205    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:26.969975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:26.969941    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:27.052113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.052078    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"]
Oct 30 01:14:27.057669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.057644    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.076154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.076125    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"]
Oct 30 01:14:27.085103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.085053    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-42d56b\"/\"rate-test-model-42d56b-kserve-self-signed-certs\""
Oct 30 01:14:27.085313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.085114    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-42d56b\"/\"default-dockercfg-k6k8h\""
Oct 30 01:14:27.087823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.087786    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-42d56b\"/\"openshift-service-ca.crt\""
Oct 30 01:14:27.087935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.087844    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-42d56b\"/\"kube-root-ca.crt\""
Oct 30 01:14:27.098095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.098066    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:27.219691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.219655    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:27.228808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.228712    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-model-cache\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.229423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.229385    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-tmp-dir\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.229555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.229446    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-home\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.229555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.229507    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d265226-b376-4d80-98b6-0f56ce3055cc-tls-certs\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.229689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.229561    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn6n4\" (UniqueName: \"kubernetes.io/projected/4d265226-b376-4d80-98b6-0f56ce3055cc-kube-api-access-vn6n4\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.229689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.229631    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-dshm\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.338018    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-model-cache\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.338225    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-tmp-dir\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.338371    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-home\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.338429    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d265226-b376-4d80-98b6-0f56ce3055cc-tls-certs\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.338483    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn6n4\" (UniqueName: \"kubernetes.io/projected/4d265226-b376-4d80-98b6-0f56ce3055cc-kube-api-access-vn6n4\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.338539    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-dshm\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.339382    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-home\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.339644    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-model-cache\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.340258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.339900    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-tmp-dir\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.353407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.353369    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d265226-b376-4d80-98b6-0f56ce3055cc-tls-certs\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.365282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.365254    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn6n4\" (UniqueName: \"kubernetes.io/projected/4d265226-b376-4d80-98b6-0f56ce3055cc-kube-api-access-vn6n4\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.375867 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.375805    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-dshm\") pod \"rate-test-model-42d56b-kserve-6489dc4585-x6fvw\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") " pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.380374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.380114    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:27.544896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.544864    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:27.706319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.706095    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm" podUID="cbc93b7e-19c5-4609-b9b7-4515f213f848" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:14:27.706986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.706953    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9" podUID="2c1f64ae-fe2e-4c69-8e98-aab96c76b832" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:14:27.761708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:27.761679    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"]
Oct 30 01:14:27.762408 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:27.762385    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d265226_b376_4d80_98b6_0f56ce3055cc.slice/crio-7469c0bd612433057ae42e1a783186a3c6a9511cfb8bf9fd2d533168b6d7bc56 WatchSource:0}: Error finding container 7469c0bd612433057ae42e1a783186a3c6a9511cfb8bf9fd2d533168b6d7bc56: Status 404 returned error can't find the container with id 7469c0bd612433057ae42e1a783186a3c6a9511cfb8bf9fd2d533168b6d7bc56
Oct 30 01:14:28.036636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.036480    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:28.049787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.049758    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:28.631332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.631277    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw" event={"ID":"4d265226-b376-4d80-98b6-0f56ce3055cc","Type":"ContainerStarted","Data":"0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c"}
Oct 30 01:14:28.631332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.631343    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw" event={"ID":"4d265226-b376-4d80-98b6-0f56ce3055cc","Type":"ContainerStarted","Data":"7469c0bd612433057ae42e1a783186a3c6a9511cfb8bf9fd2d533168b6d7bc56"}
Oct 30 01:14:28.643022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.642995    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:14:28.676827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.676775    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw" podStartSLOduration=1.676762082 podStartE2EDuration="1.676762082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:28.675431196 +0000 UTC m=+2148.746755030" watchObservedRunningTime="2026-10-30 01:14:28.676762082 +0000 UTC m=+2148.748085917"
Oct 30 01:14:28.698817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.698787    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"]
Oct 30 01:14:28.703545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.703520    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.722494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.722465    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4b973b\"/\"default-dockercfg-9v95r\""
Oct 30 01:14:28.722674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.722587    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4b973b\"/\"sub-test-model-4b973b-kserve-self-signed-certs\""
Oct 30 01:14:28.726830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.726799    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"]
Oct 30 01:14:28.778723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.778691    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4b973b\"/\"kube-root-ca.crt\""
Oct 30 01:14:28.778897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.778761    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4b973b\"/\"openshift-service-ca.crt\""
Oct 30 01:14:28.782472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.782445    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:28.860151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.860119    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-dshm\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.860363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.860306    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tls-certs\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.860416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.860378    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-model-cache\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.860506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.860482    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75q2\" (UniqueName: \"kubernetes.io/projected/c0d7d74d-6fd7-449c-b254-36f24efb49c5-kube-api-access-f75q2\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.860559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.860541    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tmp-dir\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.860605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.860591    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-home\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.967088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.967040    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-dshm\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.967311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.967193    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tls-certs\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.967427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.967350    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-model-cache\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.967427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.967388    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75q2\" (UniqueName: \"kubernetes.io/projected/c0d7d74d-6fd7-449c-b254-36f24efb49c5-kube-api-access-f75q2\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.967528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.967430    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tmp-dir\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.967528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.967472    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-home\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.968884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.967928    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-home\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.968884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.967999    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-model-cache\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.968884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.968183    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tmp-dir\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:28.984257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:28.977713    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-dshm\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:29.008271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.005945    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tls-certs\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:29.013256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.008903    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75q2\" (UniqueName: \"kubernetes.io/projected/c0d7d74d-6fd7-449c-b254-36f24efb49c5-kube-api-access-f75q2\") pod \"sub-test-model-4b973b-kserve-64df558b79-p6cl8\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") " pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:29.022968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.022943    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:29.356323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.356247    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"]
Oct 30 01:14:29.356798 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:29.356774    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0d7d74d_6fd7_449c_b254_36f24efb49c5.slice/crio-47965450c5c5b0ab453a425e52dec8c165f5247bf8a3a808feb46e4f393d982b WatchSource:0}: Error finding container 47965450c5c5b0ab453a425e52dec8c165f5247bf8a3a808feb46e4f393d982b: Status 404 returned error can't find the container with id 47965450c5c5b0ab453a425e52dec8c165f5247bf8a3a808feb46e4f393d982b
Oct 30 01:14:29.387371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.387344    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:29.567906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.567866    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:29.642036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.641932    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8" event={"ID":"c0d7d74d-6fd7-449c-b254-36f24efb49c5","Type":"ContainerStarted","Data":"13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480"}
Oct 30 01:14:29.642036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.641986    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8" event={"ID":"c0d7d74d-6fd7-449c-b254-36f24efb49c5","Type":"ContainerStarted","Data":"47965450c5c5b0ab453a425e52dec8c165f5247bf8a3a808feb46e4f393d982b"}
Oct 30 01:14:29.682209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:29.682161    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8" podStartSLOduration=1.682146673 podStartE2EDuration="1.682146673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:29.6817287 +0000 UTC m=+2149.753052564" watchObservedRunningTime="2026-10-30 01:14:29.682146673 +0000 UTC m=+2149.753470508"
Oct 30 01:14:30.023384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:30.023349    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:31.205787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:31.205755    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:32.434463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:32.434423    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:33.019365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:33.019330    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:33.273441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:33.273359    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:33.284418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:33.284391    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"]
Oct 30 01:14:33.298400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:33.298372    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bc6092-s2s9v"]
Oct 30 01:14:33.383533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:33.383487    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:33.786887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:33.786854    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:34.109297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:34.109166    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1aaeb84-f749-4cce-96a0-4dfd02b31db5" path="/var/lib/kubelet/pods/f1aaeb84-f749-4cce-96a0-4dfd02b31db5/volumes"
Oct 30 01:14:35.466446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:35.466410    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:35.739061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:35.738963    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:36.974995 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:36.974969    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:14:36.987601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:36.987578    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:14:37.040122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.040091    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"]
Oct 30 01:14:37.045443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.045410    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b5d43a-brbsn"]
Oct 30 01:14:37.072696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.072667    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc93b7e-19c5-4609-b9b7-4515f213f848-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc93b7e-19c5-4609-b9b7-4515f213f848-plugins-config-volume\") pod \"cbc93b7e-19c5-4609-b9b7-4515f213f848\" (UID: \"cbc93b7e-19c5-4609-b9b7-4515f213f848\") "
Oct 30 01:14:37.072980 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.072963    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-plugins-config-volume\") pod \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\" (UID: \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\") "
Oct 30 01:14:37.073148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.073134    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-kube-api-access-fllhj\" (UniqueName: \"kubernetes.io/projected/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-kube-api-access-fllhj\") pod \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\" (UID: \"2c1f64ae-fe2e-4c69-8e98-aab96c76b832\") "
Oct 30 01:14:37.073627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.073132    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc93b7e-19c5-4609-b9b7-4515f213f848-plugins-config-volume" pod "cbc93b7e-19c5-4609-b9b7-4515f213f848" (UID: "cbc93b7e-19c5-4609-b9b7-4515f213f848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:37.073627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.073589    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-plugins-config-volume" pod "2c1f64ae-fe2e-4c69-8e98-aab96c76b832" (UID: "2c1f64ae-fe2e-4c69-8e98-aab96c76b832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:37.073822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.073678    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc93b7e-19c5-4609-b9b7-4515f213f848-kube-api-access-qbprl\" (UniqueName: \"kubernetes.io/projected/cbc93b7e-19c5-4609-b9b7-4515f213f848-kube-api-access-qbprl\") pod \"cbc93b7e-19c5-4609-b9b7-4515f213f848\" (UID: \"cbc93b7e-19c5-4609-b9b7-4515f213f848\") "
Oct 30 01:14:37.074349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.074328    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbc93b7e-19c5-4609-b9b7-4515f213f848-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:37.074509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.074487    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:37.080368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.080339    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-kube-api-access-fllhj" pod "2c1f64ae-fe2e-4c69-8e98-aab96c76b832" (UID: "2c1f64ae-fe2e-4c69-8e98-aab96c76b832"). InnerVolumeSpecName "kube-api-access-fllhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:37.082247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.082205    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc93b7e-19c5-4609-b9b7-4515f213f848-kube-api-access-qbprl" pod "cbc93b7e-19c5-4609-b9b7-4515f213f848" (UID: "cbc93b7e-19c5-4609-b9b7-4515f213f848"). InnerVolumeSpecName "kube-api-access-qbprl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:37.082749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.082721    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:37.175420 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.175342    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fllhj\" (UniqueName: \"kubernetes.io/projected/2c1f64ae-fe2e-4c69-8e98-aab96c76b832-kube-api-access-fllhj\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:37.175420 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.175371    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbprl\" (UniqueName: \"kubernetes.io/projected/cbc93b7e-19c5-4609-b9b7-4515f213f848-kube-api-access-qbprl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:37.272633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.272594    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:37.381438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.381395    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:37.394083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.394057    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:37.708442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.708409    4576 generic.go:356] "Generic (PLEG): container finished" podID="cbc93b7e-19c5-4609-b9b7-4515f213f848" containerID="e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522" exitCode=0
Oct 30 01:14:37.708597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.708460    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"
Oct 30 01:14:37.708597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.708493    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm" event={"ID":"cbc93b7e-19c5-4609-b9b7-4515f213f848","Type":"ContainerDied","Data":"e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522"}
Oct 30 01:14:37.708597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.708546    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm" event={"ID":"cbc93b7e-19c5-4609-b9b7-4515f213f848","Type":"ContainerDied","Data":"2fec56eead553270fe2ab6539d927830466eb062809abb8fc8fc036f096ad96a"}
Oct 30 01:14:37.708597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.708569    4576 scope.go:122] "RemoveContainer" containerID="e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522"
Oct 30 01:14:37.710125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.710100    4576 generic.go:356] "Generic (PLEG): container finished" podID="2c1f64ae-fe2e-4c69-8e98-aab96c76b832" containerID="79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a" exitCode=0
Oct 30 01:14:37.710261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.710152    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"
Oct 30 01:14:37.710261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.710192    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9" event={"ID":"2c1f64ae-fe2e-4c69-8e98-aab96c76b832","Type":"ContainerDied","Data":"79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a"}
Oct 30 01:14:37.710389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.710258    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9" event={"ID":"2c1f64ae-fe2e-4c69-8e98-aab96c76b832","Type":"ContainerDied","Data":"314d9cab6c79e56f1555667fa11d7ba0fe53644473c7cd31e32888266d36d066"}
Oct 30 01:14:37.722656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.722632    4576 scope.go:122] "RemoveContainer" containerID="e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522"
Oct 30 01:14:37.722934 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:37.722915    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522\": container with ID starting with e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522 not found: ID does not exist" containerID="e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522"
Oct 30 01:14:37.722992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.722943    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522"} err="failed to get container status \"e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522\": rpc error: code = NotFound desc = could not find container \"e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522\": container with ID starting with e134efae84cf9ede2dd2aeeee79f018c3c5af97bac9dd620886e79de51253522 not found: ID does not exist"
Oct 30 01:14:37.722992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.722961    4576 scope.go:122] "RemoveContainer" containerID="79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a"
Oct 30 01:14:37.726571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.726552    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:14:37.733505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.733488    4576 scope.go:122] "RemoveContainer" containerID="79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a"
Oct 30 01:14:37.733785 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:37.733768    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a\": container with ID starting with 79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a not found: ID does not exist" containerID="79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a"
Oct 30 01:14:37.733831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.733792    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a"} err="failed to get container status \"79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a\": rpc error: code = NotFound desc = could not find container \"79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a\": container with ID starting with 79569e148991f23752fad382eb550afbaca93824f74fe1ac3ff85c85a2f5fa6a not found: ID does not exist"
Oct 30 01:14:37.766827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.766798    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"]
Oct 30 01:14:37.778527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.778502    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc6092-57cb6d584c-ffhmm"]
Oct 30 01:14:37.815097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.815073    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"]
Oct 30 01:14:37.819918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:37.819892    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc6092-6c5c64656b-jfhb9"]
Oct 30 01:14:38.109790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:38.109746    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1f64ae-fe2e-4c69-8e98-aab96c76b832" path="/var/lib/kubelet/pods/2c1f64ae-fe2e-4c69-8e98-aab96c76b832/volumes"
Oct 30 01:14:38.110280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:38.110222    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e57499-5390-46c8-9989-c866926286c4" path="/var/lib/kubelet/pods/88e57499-5390-46c8-9989-c866926286c4/volumes"
Oct 30 01:14:38.110680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:38.110663    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc93b7e-19c5-4609-b9b7-4515f213f848" path="/var/lib/kubelet/pods/cbc93b7e-19c5-4609-b9b7-4515f213f848/volumes"
Oct 30 01:14:38.647058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:38.647026    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:39.023856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:39.023825    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:39.036988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:39.036964    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:39.746583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:39.746548    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:39.750654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:39.750631    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:14:39.873243 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:39.873200    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:40.045318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:40.045213    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"]
Oct 30 01:14:40.045510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:40.045475    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy" containerID="cri-o://30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d" gracePeriod=30
Oct 30 01:14:40.101303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:40.100138    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:40.676046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:40.676011    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:40.876379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:40.876335    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:40.937375 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:40.937350    4576 scope.go:122] "RemoveContainer" containerID="a1d82415421c45c0991045fe8ae9b9afb9444e7061b241ca0d60e53864f86f62"
Oct 30 01:14:41.096737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.096705    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:41.231418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.231325    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:41.250928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.250871    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"]
Oct 30 01:14:41.251820 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.251797    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2c1f64ae-fe2e-4c69-8e98-aab96c76b832" containerName="payload-processing"
Oct 30 01:14:41.251820 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.251820    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1f64ae-fe2e-4c69-8e98-aab96c76b832" containerName="payload-processing"
Oct 30 01:14:41.252006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.251852    4576 cpu_manager.go:414] "containerMap: removing container" podUID="cbc93b7e-19c5-4609-b9b7-4515f213f848" containerName="payload-pre-processing"
Oct 30 01:14:41.252006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.251861    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc93b7e-19c5-4609-b9b7-4515f213f848" containerName="payload-pre-processing"
Oct 30 01:14:41.252105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.252007    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1f64ae-fe2e-4c69-8e98-aab96c76b832" containerName="payload-processing"
Oct 30 01:14:41.252105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.252023    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc93b7e-19c5-4609-b9b7-4515f213f848" containerName="payload-pre-processing"
Oct 30 01:14:41.262282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.259016    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.289701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.289662    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-cb31e3-openshift-default-dockercfg-cgb6r\""
Oct 30 01:14:41.293983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.293952    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"]
Oct 30 01:14:41.349626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.349495    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.349822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.349625    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-credential-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.349822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.349719    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istiod-ca-cert\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.349822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.349795    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-token\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.349958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.349830    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-podinfo\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.349958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.349881    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-certs\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.349958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.349945    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjt9\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-kube-api-access-npjt9\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.350067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.349990    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-envoy\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.350067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.350018    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-data\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.449260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.448278    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.451884    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-credential-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.452021    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istiod-ca-cert\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.452119    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-token\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.452178    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-podinfo\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.452281    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-certs\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.452384    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjt9\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-kube-api-access-npjt9\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.452430    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-envoy\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.452553    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-data\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.452686    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.453877    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.453976    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-credential-socket\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.455209    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-data\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.456409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.455813    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istiod-ca-cert\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.457184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.456561    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-certs\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.462549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.462491    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-envoy\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.480931 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.480515    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-token\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.486297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.483906    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjt9\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-kube-api-access-npjt9\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.491143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.491109    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-podinfo\") pod \"e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") " pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.591276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.591227    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:41.613347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.613292    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:41.935914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.935886    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"]
Oct 30 01:14:41.936475 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:41.936450    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc90602_3cba_4f39_95ff_d46de1dd32f0.slice/crio-e63185d0afcdff8493a11fa0af27d0c2b26d3e6fd733b36fe9bf0cbdce3ac496 WatchSource:0}: Error finding container e63185d0afcdff8493a11fa0af27d0c2b26d3e6fd733b36fe9bf0cbdce3ac496: Status 404 returned error can't find the container with id e63185d0afcdff8493a11fa0af27d0c2b26d3e6fd733b36fe9bf0cbdce3ac496
Oct 30 01:14:41.939216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.939183    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:41.939331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.939273    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:41.939331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:41.939303    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:42.482288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:42.482260    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:42.764997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:42.764895    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" event={"ID":"9cc90602-3cba-4f39-95ff-d46de1dd32f0","Type":"ContainerStarted","Data":"191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a"}
Oct 30 01:14:42.764997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:42.764959    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" event={"ID":"9cc90602-3cba-4f39-95ff-d46de1dd32f0","Type":"ContainerStarted","Data":"e63185d0afcdff8493a11fa0af27d0c2b26d3e6fd733b36fe9bf0cbdce3ac496"}
Oct 30 01:14:42.841388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:42.841317    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" podStartSLOduration=1.841293447 podStartE2EDuration="1.841293447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:42.824662615 +0000 UTC m=+2162.895986472" watchObservedRunningTime="2026-10-30 01:14:42.841293447 +0000 UTC m=+2162.912617286"
Oct 30 01:14:43.591634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.591594    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:43.599084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.599059    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:43.779708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.779676    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:43.780815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.780793    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:14:43.827878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.827836    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"]
Oct 30 01:14:43.834699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.834655    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.860645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.860451    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ca324d-openshift-default-dockercfg-w2cst\""
Oct 30 01:14:43.869541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.869505    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"]
Oct 30 01:14:43.899423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.899367    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-credential-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.899634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.899550    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.899634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.899611    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-token\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.899767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.899674    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba194d2d-7baf-4923-8084-e551b8e2125f-istiod-ca-cert\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.899767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.899753    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-data\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.899957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.899932    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-envoy\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.900078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.900058    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzltp\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-kube-api-access-wzltp\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.900141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.900128    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-certs\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:43.900207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:43.900166    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-podinfo\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.001755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.001713    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-credential-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.001966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.001800    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.001966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.001833    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-token\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.001966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.001867    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba194d2d-7baf-4923-8084-e551b8e2125f-istiod-ca-cert\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.001966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.001923    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-data\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002010    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-envoy\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002078    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzltp\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-kube-api-access-wzltp\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002131    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-certs\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002167    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-podinfo\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002193    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-credential-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002459    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-socket\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002591    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-data\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002738    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba194d2d-7baf-4923-8084-e551b8e2125f-istiod-ca-cert\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.002845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.002824    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-certs\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.004533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.004511    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-envoy\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.005260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.005225    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-podinfo\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.022740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.022704    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-token\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.023179 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.023158    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzltp\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-kube-api-access-wzltp\") pod \"e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") " pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.201929 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.201896    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:44.381189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.381139    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:44.519545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.519515    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"]
Oct 30 01:14:44.519966 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:44.519938    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba194d2d_7baf_4923_8084_e551b8e2125f.slice/crio-8692e37bacb9d8885992593ce5ab332b95e96bcec8ba402eeeadcee97ad3719b WatchSource:0}: Error finding container 8692e37bacb9d8885992593ce5ab332b95e96bcec8ba402eeeadcee97ad3719b: Status 404 returned error can't find the container with id 8692e37bacb9d8885992593ce5ab332b95e96bcec8ba402eeeadcee97ad3719b
Oct 30 01:14:44.522821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.522788    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:44.522901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.522854    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:44.522947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.522903    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:14:44.792736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.792626    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" event={"ID":"ba194d2d-7baf-4923-8084-e551b8e2125f","Type":"ContainerStarted","Data":"14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125"}
Oct 30 01:14:44.792736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.792695    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" event={"ID":"ba194d2d-7baf-4923-8084-e551b8e2125f","Type":"ContainerStarted","Data":"8692e37bacb9d8885992593ce5ab332b95e96bcec8ba402eeeadcee97ad3719b"}
Oct 30 01:14:44.839371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:44.839322    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podStartSLOduration=1.839307099 podStartE2EDuration="1.839307099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:44.835685356 +0000 UTC m=+2164.907009216" watchObservedRunningTime="2026-10-30 01:14:44.839307099 +0000 UTC m=+2164.910630943"
Oct 30 01:14:45.202524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.202488    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:45.204847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.204814    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:14:45.205017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.204878    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 01:14:45.247434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.247377    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:45.334753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.334727    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:14:45.372216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.372187    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.425900    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-credential-socket\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.425984    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istiod-ca-cert\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.426066    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-podinfo\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.426104    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-envoy\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.426150    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-socket\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.426175    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-token\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-token\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.426215    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-certs\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.426397    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-data\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.427193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.426443    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-kube-api-access-7pscl\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-kube-api-access-7pscl\") pod \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\" (UID: \"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1\") "
Oct 30 01:14:45.428585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.428289    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-credential-socket" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:45.428585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.428304    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istiod-ca-cert" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:45.429435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.429009    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-data" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:45.429435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.429393    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-socket" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:45.429736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.429708    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-kube-api-access-7pscl" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "kube-api-access-7pscl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:45.430021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.429996    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-certs" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:45.430711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.430679    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-envoy" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:14:45.433706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.433606    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-podinfo" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:14:45.434990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.434963    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-token" pod "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" (UID: "5cc735cf-9b64-4fcc-9f94-19fc59baf9c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:45.528923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528884    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.528923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528918    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pscl\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-kube-api-access-7pscl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.528923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528928    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.529165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528939    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.529165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528950    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.529165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528958    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.529165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528965    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.529165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528972    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.529165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.528979    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:45.803687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.803650    4576 generic.go:356] "Generic (PLEG): container finished" podID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerID="30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d" exitCode=0
Oct 30 01:14:45.804255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.803759    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"
Oct 30 01:14:45.804255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.803763    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" event={"ID":"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1","Type":"ContainerDied","Data":"30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d"}
Oct 30 01:14:45.804255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.803833    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s" event={"ID":"5cc735cf-9b64-4fcc-9f94-19fc59baf9c1","Type":"ContainerDied","Data":"517c795daa1a0a345ed68399e4636ffed79cc9c5886ee7cf16fe676e864b0643"}
Oct 30 01:14:45.804255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.803866    4576 scope.go:122] "RemoveContainer" containerID="30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d"
Oct 30 01:14:45.815734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.815715    4576 scope.go:122] "RemoveContainer" containerID="30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d"
Oct 30 01:14:45.816006 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:45.815986    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d\": container with ID starting with 30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d not found: ID does not exist" containerID="30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d"
Oct 30 01:14:45.816059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.816015    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d"} err="failed to get container status \"30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d\": rpc error: code = NotFound desc = could not find container \"30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d\": container with ID starting with 30b6ca0df73de6ca986061b99d5cf287dc0cd09822ede30608364e8b82e1aa6d not found: ID does not exist"
Oct 30 01:14:45.861329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.861303    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"]
Oct 30 01:14:45.896719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:45.896672    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bc6092-openshift-default-5db4ddf765-7c25s"]
Oct 30 01:14:46.110868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:46.110789    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" path="/var/lib/kubelet/pods/5cc735cf-9b64-4fcc-9f94-19fc59baf9c1/volumes"
Oct 30 01:14:46.202893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:46.202854    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:14:46.203080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:46.202943    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 01:14:46.408436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:46.408347    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:47.020877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:47.020833    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:47.136762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:47.136727    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:47.203639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:47.203147    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:14:47.203639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:47.203222    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 01:14:48.185287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:48.185243    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:48.203074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:48.203038    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:14:48.203300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:48.203100    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 01:14:48.404946 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:14:48.404897    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:14:49.202461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:49.202427    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:14:49.202860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:49.202481    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 01:14:49.300218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:49.300170    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:49.991560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:49.991528    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:50.204638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:50.204599    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:14:50.205151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:50.204655    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 01:14:51.202961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:51.202908    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 01:14:51.203199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:51.202977    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 01:14:52.069128 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:52.069084    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:52.208354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:52.208325    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:52.208609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:52.208584    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:52.209439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:52.209415    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:14:53.607392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:53.607363    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:54.337727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.337678    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"]
Oct 30 01:14:54.338498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.338471    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy"
Oct 30 01:14:54.338498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.338497    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy"
Oct 30 01:14:54.338696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.338660    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc735cf-9b64-4fcc-9f94-19fc59baf9c1" containerName="istio-proxy"
Oct 30 01:14:54.343803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.343782    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.357588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.357554    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"]
Oct 30 01:14:54.367717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.367691    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:14:54.367853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.367735    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:14:54.391872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.391833    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.392083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.392063    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snlbq\" (UniqueName: \"kubernetes.io/projected/0780e8f4-f5c6-4511-bc88-5166f857df8d-kube-api-access-snlbq\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.392318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.392298    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.493193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.493157    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.493429 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.493255    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.493429 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.493341    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snlbq\" (UniqueName: \"kubernetes.io/projected/0780e8f4-f5c6-4511-bc88-5166f857df8d-kube-api-access-snlbq\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.493954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.493928    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.495788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.495767    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.513778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.513754    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snlbq\" (UniqueName: \"kubernetes.io/projected/0780e8f4-f5c6-4511-bc88-5166f857df8d-kube-api-access-snlbq\") pod \"maas-api-revoke-sub-fc9e53ae-7b84t\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.659270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.659180    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:54.966527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:54.966501    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"]
Oct 30 01:14:54.967758 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:14:54.967733    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0780e8f4_f5c6_4511_bc88_5166f857df8d.slice/crio-cc34e93d7ba3470e57159f2a98c1f3b54d0b8d906bbdcf76399fcb2a52cb65be WatchSource:0}: Error finding container cc34e93d7ba3470e57159f2a98c1f3b54d0b8d906bbdcf76399fcb2a52cb65be: Status 404 returned error can't find the container with id cc34e93d7ba3470e57159f2a98c1f3b54d0b8d906bbdcf76399fcb2a52cb65be
Oct 30 01:14:55.091144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:55.091104    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:55.879540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:55.879507    4576 generic.go:356] "Generic (PLEG): container finished" podID="0780e8f4-f5c6-4511-bc88-5166f857df8d" containerID="522833fb268db7a739d730a9fd335567d931c2bc2baa67fb11e9d7a05703c116" exitCode=0
Oct 30 01:14:55.879950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:55.879583    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t" event={"ID":"0780e8f4-f5c6-4511-bc88-5166f857df8d","Type":"ContainerDied","Data":"522833fb268db7a739d730a9fd335567d931c2bc2baa67fb11e9d7a05703c116"}
Oct 30 01:14:55.879950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:55.879657    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t" event={"ID":"0780e8f4-f5c6-4511-bc88-5166f857df8d","Type":"ContainerStarted","Data":"cc34e93d7ba3470e57159f2a98c1f3b54d0b8d906bbdcf76399fcb2a52cb65be"}
Oct 30 01:14:56.693206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:56.693167    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:57.049910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.049883    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:57.129525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.129491    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-openshift-ca\") pod \"0780e8f4-f5c6-4511-bc88-5166f857df8d\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") "
Oct 30 01:14:57.129738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.129583    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0780e8f4-f5c6-4511-bc88-5166f857df8d-kube-api-access-snlbq\" (UniqueName: \"kubernetes.io/projected/0780e8f4-f5c6-4511-bc88-5166f857df8d-kube-api-access-snlbq\") pod \"0780e8f4-f5c6-4511-bc88-5166f857df8d\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") "
Oct 30 01:14:57.129843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.129764    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-xks-ca\") pod \"0780e8f4-f5c6-4511-bc88-5166f857df8d\" (UID: \"0780e8f4-f5c6-4511-bc88-5166f857df8d\") "
Oct 30 01:14:57.129955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.129932    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-openshift-ca" pod "0780e8f4-f5c6-4511-bc88-5166f857df8d" (UID: "0780e8f4-f5c6-4511-bc88-5166f857df8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:57.130910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.130884    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.132181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.132151    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0780e8f4-f5c6-4511-bc88-5166f857df8d-kube-api-access-snlbq" pod "0780e8f4-f5c6-4511-bc88-5166f857df8d" (UID: "0780e8f4-f5c6-4511-bc88-5166f857df8d"). InnerVolumeSpecName "kube-api-access-snlbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:57.132627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.132604    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-xks-ca" pod "0780e8f4-f5c6-4511-bc88-5166f857df8d" (UID: "0780e8f4-f5c6-4511-bc88-5166f857df8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:57.232088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.231998    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snlbq\" (UniqueName: \"kubernetes.io/projected/0780e8f4-f5c6-4511-bc88-5166f857df8d-kube-api-access-snlbq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.232088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.232032    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0780e8f4-f5c6-4511-bc88-5166f857df8d-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:14:57.896918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.896881    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"
Oct 30 01:14:57.897140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.896930    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t" event={"ID":"0780e8f4-f5c6-4511-bc88-5166f857df8d","Type":"ContainerDied","Data":"cc34e93d7ba3470e57159f2a98c1f3b54d0b8d906bbdcf76399fcb2a52cb65be"}
Oct 30 01:14:57.897140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:57.896978    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc34e93d7ba3470e57159f2a98c1f3b54d0b8d906bbdcf76399fcb2a52cb65be"
Oct 30 01:14:58.256361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:58.256324    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:14:59.935719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:14:59.935685    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:00.040760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.040721    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"]
Oct 30 01:15:00.050495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.050462    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:00.050943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.050917    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e789959-j4xw8"]
Oct 30 01:15:00.109530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.109496    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a2f886-c8aa-455a-af8e-f0a35fae0c8c" path="/var/lib/kubelet/pods/97a2f886-c8aa-455a-af8e-f0a35fae0c8c/volumes"
Oct 30 01:15:00.142294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.142254    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"]
Oct 30 01:15:00.143209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.143171    4576 cpu_manager.go:414] "containerMap: removing container" podUID="0780e8f4-f5c6-4511-bc88-5166f857df8d" containerName="revoke-keys"
Oct 30 01:15:00.143353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.143213    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0780e8f4-f5c6-4511-bc88-5166f857df8d" containerName="revoke-keys"
Oct 30 01:15:00.143420 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.143409    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="0780e8f4-f5c6-4511-bc88-5166f857df8d" containerName="revoke-keys"
Oct 30 01:15:00.148945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.148921    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"]
Oct 30 01:15:00.149095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.149045    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:00.153177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.153155    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"]
Oct 30 01:15:00.153304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.153226    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:00.157297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.157271    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"]
Oct 30 01:15:00.157431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.157330    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:00.161540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.161521    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:00.167171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.167136    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"]
Oct 30 01:15:00.171678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.171647    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"]
Oct 30 01:15:00.181648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.181627    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:15:00.185116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.185094    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"]
Oct 30 01:15:00.191311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.191209    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"]
Oct 30 01:15:00.195263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.195244    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"]
Oct 30 01:15:00.204193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.204173    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"]
Oct 30 01:15:00.206009 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.205878    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:00.210324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.210207    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:00.237489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.237458    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"]
Oct 30 01:15:00.238736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.238711    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:00.244297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.244223    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"]
Oct 30 01:15:00.244447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.244322    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"]
Oct 30 01:15:00.244447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.244377    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:00.252469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.252442    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"]
Oct 30 01:15:00.255156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.255133    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:00.260604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.260579    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"]
Oct 30 01:15:00.260815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.260799    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"]
Oct 30 01:15:00.267466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.266400    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:00.268655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.268634    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"]
Oct 30 01:15:00.271530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.271502    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:00.280074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.280053    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"]
Oct 30 01:15:00.280328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.280311    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"]
Oct 30 01:15:00.280482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.280468    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"]
Oct 30 01:15:00.280614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.280601    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"]
Oct 30 01:15:00.280801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.280788    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:00.281913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.281891    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bb9\" (UniqueName: \"kubernetes.io/projected/54cd525e-0953-419a-9f8b-f9b3c1318918-kube-api-access-46bb9\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8\" (UID: \"54cd525e-0953-419a-9f8b-f9b3c1318918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:00.282080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.282060    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbwv\" (UniqueName: \"kubernetes.io/projected/e7290ded-d2a3-49d1-b712-dc57bac30f01-kube-api-access-rbbwv\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp\" (UID: \"e7290ded-d2a3-49d1-b712-dc57bac30f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:00.282416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.282394    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e583bfd6-926a-47a6-9885-74675eac92d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl\" (UID: \"e583bfd6-926a-47a6-9885-74675eac92d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:00.282575 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.282560    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7290ded-d2a3-49d1-b712-dc57bac30f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp\" (UID: \"e7290ded-d2a3-49d1-b712-dc57bac30f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:00.282776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.282758    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cxh\" (UniqueName: \"kubernetes.io/projected/497a3e39-1661-457e-8fc4-073c263f2f2c-kube-api-access-d8cxh\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b\" (UID: \"497a3e39-1661-457e-8fc4-073c263f2f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:00.282896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.282879    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbxs\" (UniqueName: \"kubernetes.io/projected/fff10f99-a41c-46a1-9dc3-5824e57cf03b-kube-api-access-vvbxs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww\" (UID: \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:00.283381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.283365    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpv46\" (UniqueName: \"kubernetes.io/projected/e583bfd6-926a-47a6-9885-74675eac92d0-kube-api-access-bpv46\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl\" (UID: \"e583bfd6-926a-47a6-9885-74675eac92d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:00.283503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.283493    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff10f99-a41c-46a1-9dc3-5824e57cf03b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww\" (UID: \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:00.283604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.283594    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/497a3e39-1661-457e-8fc4-073c263f2f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b\" (UID: \"497a3e39-1661-457e-8fc4-073c263f2f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:00.283689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.283679    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cd525e-0953-419a-9f8b-f9b3c1318918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8\" (UID: \"54cd525e-0953-419a-9f8b-f9b3c1318918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:00.290996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.290977    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"]
Oct 30 01:15:00.385219 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385184    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7290ded-d2a3-49d1-b712-dc57bac30f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp\" (UID: \"e7290ded-d2a3-49d1-b712-dc57bac30f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:00.385467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385268    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdtz\" (UniqueName: \"kubernetes.io/projected/2d98a290-3f4a-4467-ab4d-623a42397144-kube-api-access-6vdtz\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm\" (UID: \"2d98a290-3f4a-4467-ab4d-623a42397144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:00.385467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385292    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhq6m\" (UniqueName: \"kubernetes.io/projected/818ea717-759d-4697-9367-6ede754cdc48-kube-api-access-hhq6m\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx\" (UID: \"818ea717-759d-4697-9367-6ede754cdc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:00.385467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385310    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cxh\" (UniqueName: \"kubernetes.io/projected/497a3e39-1661-457e-8fc4-073c263f2f2c-kube-api-access-d8cxh\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b\" (UID: \"497a3e39-1661-457e-8fc4-073c263f2f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:00.385467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385332    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbxs\" (UniqueName: \"kubernetes.io/projected/fff10f99-a41c-46a1-9dc3-5824e57cf03b-kube-api-access-vvbxs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww\" (UID: \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:00.385467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385398    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zm5\" (UniqueName: \"kubernetes.io/projected/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-kube-api-access-j6zm5\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l\" (UID: \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:00.385772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385526    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv46\" (UniqueName: \"kubernetes.io/projected/e583bfd6-926a-47a6-9885-74675eac92d0-kube-api-access-bpv46\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl\" (UID: \"e583bfd6-926a-47a6-9885-74675eac92d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:00.385772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385571    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818ea717-759d-4697-9367-6ede754cdc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx\" (UID: \"818ea717-759d-4697-9367-6ede754cdc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:00.385772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385610    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff10f99-a41c-46a1-9dc3-5824e57cf03b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww\" (UID: \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:00.385772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385706    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpbr\" (UniqueName: \"kubernetes.io/projected/58404bec-1cc1-46cc-a5e7-a6994b9c1163-kube-api-access-jmpbr\") pod \"maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v\" (UID: \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:00.385772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385742    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/497a3e39-1661-457e-8fc4-073c263f2f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b\" (UID: \"497a3e39-1661-457e-8fc4-073c263f2f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:00.386040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385797    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwbk\" (UniqueName: \"kubernetes.io/projected/36763be6-dedf-4cc7-b154-430d488e2b33-kube-api-access-lmwbk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf\" (UID: \"36763be6-dedf-4cc7-b154-430d488e2b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:00.386040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385843    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cd525e-0953-419a-9f8b-f9b3c1318918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8\" (UID: \"54cd525e-0953-419a-9f8b-f9b3c1318918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:00.386040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385926    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36763be6-dedf-4cc7-b154-430d488e2b33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf\" (UID: \"36763be6-dedf-4cc7-b154-430d488e2b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:00.386040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.385976    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bb9\" (UniqueName: \"kubernetes.io/projected/54cd525e-0953-419a-9f8b-f9b3c1318918-kube-api-access-46bb9\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8\" (UID: \"54cd525e-0953-419a-9f8b-f9b3c1318918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:00.386040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386001    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58404bec-1cc1-46cc-a5e7-a6994b9c1163-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v\" (UID: \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:00.386040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386037    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbwv\" (UniqueName: \"kubernetes.io/projected/e7290ded-d2a3-49d1-b712-dc57bac30f01-kube-api-access-rbbwv\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp\" (UID: \"e7290ded-d2a3-49d1-b712-dc57bac30f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:00.386381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386061    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7290ded-d2a3-49d1-b712-dc57bac30f01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp\" (UID: \"e7290ded-d2a3-49d1-b712-dc57bac30f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:00.386381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386070    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98a290-3f4a-4467-ab4d-623a42397144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm\" (UID: \"2d98a290-3f4a-4467-ab4d-623a42397144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:00.386381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386152    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hg4\" (UniqueName: \"kubernetes.io/projected/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-kube-api-access-j4hg4\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85\" (UID: \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:00.386381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386284    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85\" (UID: \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:00.386381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386320    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l\" (UID: \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:00.386381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386350    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e583bfd6-926a-47a6-9885-74675eac92d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl\" (UID: \"e583bfd6-926a-47a6-9885-74675eac92d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:00.386624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386423    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff10f99-a41c-46a1-9dc3-5824e57cf03b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww\" (UID: \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:00.386624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386507    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cd525e-0953-419a-9f8b-f9b3c1318918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8\" (UID: \"54cd525e-0953-419a-9f8b-f9b3c1318918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:00.386624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386579    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/497a3e39-1661-457e-8fc4-073c263f2f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b\" (UID: \"497a3e39-1661-457e-8fc4-073c263f2f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:00.386922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.386905    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e583bfd6-926a-47a6-9885-74675eac92d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl\" (UID: \"e583bfd6-926a-47a6-9885-74675eac92d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:00.406600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.406561    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpv46\" (UniqueName: \"kubernetes.io/projected/e583bfd6-926a-47a6-9885-74675eac92d0-kube-api-access-bpv46\") pod \"maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl\" (UID: \"e583bfd6-926a-47a6-9885-74675eac92d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:00.406789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.406765    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cxh\" (UniqueName: \"kubernetes.io/projected/497a3e39-1661-457e-8fc4-073c263f2f2c-kube-api-access-d8cxh\") pod \"maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b\" (UID: \"497a3e39-1661-457e-8fc4-073c263f2f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:00.406845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.406776    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbxs\" (UniqueName: \"kubernetes.io/projected/fff10f99-a41c-46a1-9dc3-5824e57cf03b-kube-api-access-vvbxs\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww\" (UID: \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:00.406845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.406776    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bb9\" (UniqueName: \"kubernetes.io/projected/54cd525e-0953-419a-9f8b-f9b3c1318918-kube-api-access-46bb9\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8\" (UID: \"54cd525e-0953-419a-9f8b-f9b3c1318918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:00.408516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.408498    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbwv\" (UniqueName: \"kubernetes.io/projected/e7290ded-d2a3-49d1-b712-dc57bac30f01-kube-api-access-rbbwv\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp\" (UID: \"e7290ded-d2a3-49d1-b712-dc57bac30f01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:00.485930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.485845    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:00.487534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.487504    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85\" (UID: \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:00.487680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.487544    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l\" (UID: \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:00.487757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.487725    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdtz\" (UniqueName: \"kubernetes.io/projected/2d98a290-3f4a-4467-ab4d-623a42397144-kube-api-access-6vdtz\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm\" (UID: \"2d98a290-3f4a-4467-ab4d-623a42397144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:00.487827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.487765    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhq6m\" (UniqueName: \"kubernetes.io/projected/818ea717-759d-4697-9367-6ede754cdc48-kube-api-access-hhq6m\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx\" (UID: \"818ea717-759d-4697-9367-6ede754cdc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:00.487827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.487799    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zm5\" (UniqueName: \"kubernetes.io/projected/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-kube-api-access-j6zm5\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l\" (UID: \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:00.487962 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.487884    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818ea717-759d-4697-9367-6ede754cdc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx\" (UID: \"818ea717-759d-4697-9367-6ede754cdc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:00.487962 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.487953    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpbr\" (UniqueName: \"kubernetes.io/projected/58404bec-1cc1-46cc-a5e7-a6994b9c1163-kube-api-access-jmpbr\") pod \"maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v\" (UID: \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:00.488123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488000    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwbk\" (UniqueName: \"kubernetes.io/projected/36763be6-dedf-4cc7-b154-430d488e2b33-kube-api-access-lmwbk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf\" (UID: \"36763be6-dedf-4cc7-b154-430d488e2b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:00.488123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488052    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36763be6-dedf-4cc7-b154-430d488e2b33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf\" (UID: \"36763be6-dedf-4cc7-b154-430d488e2b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:00.488123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488093    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58404bec-1cc1-46cc-a5e7-a6994b9c1163-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v\" (UID: \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:00.488338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488134    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98a290-3f4a-4467-ab4d-623a42397144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm\" (UID: \"2d98a290-3f4a-4467-ab4d-623a42397144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:00.488338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488168    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hg4\" (UniqueName: \"kubernetes.io/projected/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-kube-api-access-j4hg4\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85\" (UID: \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:00.488451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488363    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l\" (UID: \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:00.488514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488469    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85\" (UID: \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:00.488639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488620    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818ea717-759d-4697-9367-6ede754cdc48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx\" (UID: \"818ea717-759d-4697-9367-6ede754cdc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:00.488973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488949    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58404bec-1cc1-46cc-a5e7-a6994b9c1163-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v\" (UID: \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:00.489022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488950    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36763be6-dedf-4cc7-b154-430d488e2b33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf\" (UID: \"36763be6-dedf-4cc7-b154-430d488e2b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:00.489022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.488959    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98a290-3f4a-4467-ab4d-623a42397144-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm\" (UID: \"2d98a290-3f4a-4467-ab4d-623a42397144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:00.495461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.495442    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:00.509692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.509662    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdtz\" (UniqueName: \"kubernetes.io/projected/2d98a290-3f4a-4467-ab4d-623a42397144-kube-api-access-6vdtz\") pod \"maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm\" (UID: \"2d98a290-3f4a-4467-ab4d-623a42397144\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:00.510735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.510159    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zm5\" (UniqueName: \"kubernetes.io/projected/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-kube-api-access-j6zm5\") pod \"maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l\" (UID: \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:00.510735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.510689    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwbk\" (UniqueName: \"kubernetes.io/projected/36763be6-dedf-4cc7-b154-430d488e2b33-kube-api-access-lmwbk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf\" (UID: \"36763be6-dedf-4cc7-b154-430d488e2b33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:00.511115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.511091    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhq6m\" (UniqueName: \"kubernetes.io/projected/818ea717-759d-4697-9367-6ede754cdc48-kube-api-access-hhq6m\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx\" (UID: \"818ea717-759d-4697-9367-6ede754cdc48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:00.513534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.513508    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:00.513874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.513849    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hg4\" (UniqueName: \"kubernetes.io/projected/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-kube-api-access-j4hg4\") pod \"maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85\" (UID: \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:00.514029 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.514007    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpbr\" (UniqueName: \"kubernetes.io/projected/58404bec-1cc1-46cc-a5e7-a6994b9c1163-kube-api-access-jmpbr\") pod \"maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v\" (UID: \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:00.527267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.527218    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:00.545655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.545347    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:00.580980 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.566783    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:00.594193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.593969    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:00.608411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.608332    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:00.620980 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.620953    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:00.630527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.628854    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:00.645097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.644597    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:00.845973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.845722    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"]
Oct 30 01:15:00.942762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:00.942662    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" event={"ID":"497a3e39-1661-457e-8fc4-073c263f2f2c","Type":"ContainerStarted","Data":"f3d824914a6e0a3eede7325fdef84969744b7971f7d3345c3cae2729b2036d5d"}
Oct 30 01:15:01.259561 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:01.258496    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode583bfd6_926a_47a6_9885_74675eac92d0.slice/crio-fe967cd171539f18a1229fdfb05f1915f4038dd6977c77e869f3aa8152049eb9 WatchSource:0}: Error finding container fe967cd171539f18a1229fdfb05f1915f4038dd6977c77e869f3aa8152049eb9: Status 404 returned error can't find the container with id fe967cd171539f18a1229fdfb05f1915f4038dd6977c77e869f3aa8152049eb9
Oct 30 01:15:01.269520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.268530    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"]
Oct 30 01:15:01.279784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.279655    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"]
Oct 30 01:15:01.288639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.288610    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"]
Oct 30 01:15:01.354511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.354482    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"]
Oct 30 01:15:01.422547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.422402    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"]
Oct 30 01:15:01.455005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.454982    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"]
Oct 30 01:15:01.458498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.458474    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"]
Oct 30 01:15:01.472688 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:01.472662    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cd525e_0953_419a_9f8b_f9b3c1318918.slice/crio-ebfb6ba4d593ba2e806cc3d04382970cb9058b4789f1d628a84694c0f20ac258 WatchSource:0}: Error finding container ebfb6ba4d593ba2e806cc3d04382970cb9058b4789f1d628a84694c0f20ac258: Status 404 returned error can't find the container with id ebfb6ba4d593ba2e806cc3d04382970cb9058b4789f1d628a84694c0f20ac258
Oct 30 01:15:01.473792 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:01.473769    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58404bec_1cc1_46cc_a5e7_a6994b9c1163.slice/crio-3eb29e8d9ceb926d4a2246e4157bc35496dc4fbcfe21bfbd8ea42f3ccb9e9526 WatchSource:0}: Error finding container 3eb29e8d9ceb926d4a2246e4157bc35496dc4fbcfe21bfbd8ea42f3ccb9e9526: Status 404 returned error can't find the container with id 3eb29e8d9ceb926d4a2246e4157bc35496dc4fbcfe21bfbd8ea42f3ccb9e9526
Oct 30 01:15:01.475361 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:01.475338    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e84f94a_3cd7_48ab_844d_e0605f5a16b2.slice/crio-4a49c51264b536a7aa64c70494912e61410fae55cb7206f1df2cafcb74aa25dc WatchSource:0}: Error finding container 4a49c51264b536a7aa64c70494912e61410fae55cb7206f1df2cafcb74aa25dc: Status 404 returned error can't find the container with id 4a49c51264b536a7aa64c70494912e61410fae55cb7206f1df2cafcb74aa25dc
Oct 30 01:15:01.494433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.494408    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"]
Oct 30 01:15:01.502729 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:01.502689    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36763be6_dedf_4cc7_b154_430d488e2b33.slice/crio-2e12b6255cc1d9f19fb7a6903cf57f9ae733305429ab7b72ab9b15c6d4dc54c4 WatchSource:0}: Error finding container 2e12b6255cc1d9f19fb7a6903cf57f9ae733305429ab7b72ab9b15c6d4dc54c4: Status 404 returned error can't find the container with id 2e12b6255cc1d9f19fb7a6903cf57f9ae733305429ab7b72ab9b15c6d4dc54c4
Oct 30 01:15:01.583067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.583041    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"]
Oct 30 01:15:01.593487 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:01.593458    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d98a290_3f4a_4467_ab4d_623a42397144.slice/crio-4b430f21d3e49b163059514178805815b5f9a53823a111c2c9265aacb1578c58 WatchSource:0}: Error finding container 4b430f21d3e49b163059514178805815b5f9a53823a111c2c9265aacb1578c58: Status 404 returned error can't find the container with id 4b430f21d3e49b163059514178805815b5f9a53823a111c2c9265aacb1578c58
Oct 30 01:15:01.619562 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.619540    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"]
Oct 30 01:15:01.622252 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:01.622213    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818ea717_759d_4697_9367_6ede754cdc48.slice/crio-d18654b95ed63e7fb0b6e5578430ae5636bd8a6df80a19051166f985c0f6528d WatchSource:0}: Error finding container d18654b95ed63e7fb0b6e5578430ae5636bd8a6df80a19051166f985c0f6528d: Status 404 returned error can't find the container with id d18654b95ed63e7fb0b6e5578430ae5636bd8a6df80a19051166f985c0f6528d
Oct 30 01:15:01.769921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.769893    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:01.952319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.952277    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" event={"ID":"e583bfd6-926a-47a6-9885-74675eac92d0","Type":"ContainerStarted","Data":"60376432365e8d58e0e860343a679a5a16c783b622162cbc7fb58f6f099d10c5"}
Oct 30 01:15:01.952784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.952359    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" event={"ID":"e583bfd6-926a-47a6-9885-74675eac92d0","Type":"ContainerStarted","Data":"fe967cd171539f18a1229fdfb05f1915f4038dd6977c77e869f3aa8152049eb9"}
Oct 30 01:15:01.953799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.953771    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" event={"ID":"818ea717-759d-4697-9367-6ede754cdc48","Type":"ContainerStarted","Data":"d18654b95ed63e7fb0b6e5578430ae5636bd8a6df80a19051166f985c0f6528d"}
Oct 30 01:15:01.955390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.955365    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" event={"ID":"e7290ded-d2a3-49d1-b712-dc57bac30f01","Type":"ContainerStarted","Data":"7b52b85f3f3299290ec9270803f4d58756afb19ce31607a8c81b0a552035a801"}
Oct 30 01:15:01.955521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.955412    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" event={"ID":"e7290ded-d2a3-49d1-b712-dc57bac30f01","Type":"ContainerStarted","Data":"d2775b6b21de2b081d4db709e626802b4e30ecfa41c860afb996e5fee2b03126"}
Oct 30 01:15:01.957019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.956988    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" event={"ID":"fff10f99-a41c-46a1-9dc3-5824e57cf03b","Type":"ContainerStarted","Data":"9b4a0c6299fc0c7e46750d9201ee13bcf6ec39e4db942f381436c0b07975f003"}
Oct 30 01:15:01.957143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.957033    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" event={"ID":"fff10f99-a41c-46a1-9dc3-5824e57cf03b","Type":"ContainerStarted","Data":"f7b044127ee35ef1d43a737ca373a431dda058c14a210a2c8b73fd7aeb413237"}
Oct 30 01:15:01.958466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.958438    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" event={"ID":"58404bec-1cc1-46cc-a5e7-a6994b9c1163","Type":"ContainerStarted","Data":"3eb29e8d9ceb926d4a2246e4157bc35496dc4fbcfe21bfbd8ea42f3ccb9e9526"}
Oct 30 01:15:01.959856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.959823    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" event={"ID":"2d98a290-3f4a-4467-ab4d-623a42397144","Type":"ContainerStarted","Data":"4b430f21d3e49b163059514178805815b5f9a53823a111c2c9265aacb1578c58"}
Oct 30 01:15:01.960993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.960972    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" event={"ID":"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85","Type":"ContainerStarted","Data":"df97ae66020e95a5ffdc64215fbececdc73f9effc40ba871c3d35e92af9ade8b"}
Oct 30 01:15:01.962133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.962110    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" event={"ID":"54cd525e-0953-419a-9f8b-f9b3c1318918","Type":"ContainerStarted","Data":"ebfb6ba4d593ba2e806cc3d04382970cb9058b4789f1d628a84694c0f20ac258"}
Oct 30 01:15:01.963350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.963329    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" event={"ID":"36763be6-dedf-4cc7-b154-430d488e2b33","Type":"ContainerStarted","Data":"2e12b6255cc1d9f19fb7a6903cf57f9ae733305429ab7b72ab9b15c6d4dc54c4"}
Oct 30 01:15:01.964482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.964462    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" event={"ID":"5e84f94a-3cd7-48ab-844d-e0605f5a16b2","Type":"ContainerStarted","Data":"4a49c51264b536a7aa64c70494912e61410fae55cb7206f1df2cafcb74aa25dc"}
Oct 30 01:15:01.965976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.965957    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" event={"ID":"497a3e39-1661-457e-8fc4-073c263f2f2c","Type":"ContainerStarted","Data":"d3326e2526998cce24946e263f31182b89c76d8c5e8a43e56340f9a651e940ab"}
Oct 30 01:15:01.996708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:01.996664    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" podStartSLOduration=1.996650459 podStartE2EDuration="1.996650459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.992613265 +0000 UTC m=+2182.063937100" watchObservedRunningTime="2026-10-30 01:15:01.996650459 +0000 UTC m=+2182.067974296"
Oct 30 01:15:02.038769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.038714    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" podStartSLOduration=2.038700821 podStartE2EDuration="2.038700821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.037542107 +0000 UTC m=+2182.108865943" watchObservedRunningTime="2026-10-30 01:15:02.038700821 +0000 UTC m=+2182.110024655"
Oct 30 01:15:02.077593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.077521    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" podStartSLOduration=2.077504167 podStartE2EDuration="2.077504167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.076884349 +0000 UTC m=+2182.148208194" watchObservedRunningTime="2026-10-30 01:15:02.077504167 +0000 UTC m=+2182.148828003"
Oct 30 01:15:02.120633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.120582    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" podStartSLOduration=2.120568558 podStartE2EDuration="2.120568558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.117290872 +0000 UTC m=+2182.188614704" watchObservedRunningTime="2026-10-30 01:15:02.120568558 +0000 UTC m=+2182.191892394"
Oct 30 01:15:02.976531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.976439    4576 generic.go:356] "Generic (PLEG): container finished" podID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerID="ef01da02e1e539e0f3e99d7e3ae2584d618e90f80c3b904d37ab6415757dbf70" exitCode=60
Oct 30 01:15:02.976974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.976527    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" event={"ID":"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85","Type":"ContainerDied","Data":"ef01da02e1e539e0f3e99d7e3ae2584d618e90f80c3b904d37ab6415757dbf70"}
Oct 30 01:15:02.976974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.976860    4576 scope.go:122] "RemoveContainer" containerID="ef01da02e1e539e0f3e99d7e3ae2584d618e90f80c3b904d37ab6415757dbf70"
Oct 30 01:15:02.978609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.978584    4576 generic.go:356] "Generic (PLEG): container finished" podID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerID="a777989a1180eb98dde4184984d5ab8dbba47a6c8d2535e4cf9a3e9e23af5f7b" exitCode=60
Oct 30 01:15:02.978731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.978676    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" event={"ID":"54cd525e-0953-419a-9f8b-f9b3c1318918","Type":"ContainerDied","Data":"a777989a1180eb98dde4184984d5ab8dbba47a6c8d2535e4cf9a3e9e23af5f7b"}
Oct 30 01:15:02.978941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.978921    4576 scope.go:122] "RemoveContainer" containerID="a777989a1180eb98dde4184984d5ab8dbba47a6c8d2535e4cf9a3e9e23af5f7b"
Oct 30 01:15:02.985594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.985567    4576 generic.go:356] "Generic (PLEG): container finished" podID="36763be6-dedf-4cc7-b154-430d488e2b33" containerID="b6fb0b68fa410850cfbed70f46078f3e273784a8c4c5b70e80ec3f4c039453c0" exitCode=60
Oct 30 01:15:02.985754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.985715    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" event={"ID":"36763be6-dedf-4cc7-b154-430d488e2b33","Type":"ContainerDied","Data":"b6fb0b68fa410850cfbed70f46078f3e273784a8c4c5b70e80ec3f4c039453c0"}
Oct 30 01:15:02.986004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.985986    4576 scope.go:122] "RemoveContainer" containerID="b6fb0b68fa410850cfbed70f46078f3e273784a8c4c5b70e80ec3f4c039453c0"
Oct 30 01:15:02.988066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.988039    4576 generic.go:356] "Generic (PLEG): container finished" podID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerID="85290b4c1f99fba54f3ac1d609c7647ad075d72c765f5d8124c8164f8a2b275a" exitCode=60
Oct 30 01:15:02.988154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.988091    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" event={"ID":"5e84f94a-3cd7-48ab-844d-e0605f5a16b2","Type":"ContainerDied","Data":"85290b4c1f99fba54f3ac1d609c7647ad075d72c765f5d8124c8164f8a2b275a"}
Oct 30 01:15:02.988365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.988348    4576 scope.go:122] "RemoveContainer" containerID="85290b4c1f99fba54f3ac1d609c7647ad075d72c765f5d8124c8164f8a2b275a"
Oct 30 01:15:02.990405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.990383    4576 generic.go:356] "Generic (PLEG): container finished" podID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerID="d3326e2526998cce24946e263f31182b89c76d8c5e8a43e56340f9a651e940ab" exitCode=60
Oct 30 01:15:02.990505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.990446    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" event={"ID":"497a3e39-1661-457e-8fc4-073c263f2f2c","Type":"ContainerDied","Data":"d3326e2526998cce24946e263f31182b89c76d8c5e8a43e56340f9a651e940ab"}
Oct 30 01:15:02.990703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.990685    4576 scope.go:122] "RemoveContainer" containerID="d3326e2526998cce24946e263f31182b89c76d8c5e8a43e56340f9a651e940ab"
Oct 30 01:15:02.993770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.993513    4576 generic.go:356] "Generic (PLEG): container finished" podID="e583bfd6-926a-47a6-9885-74675eac92d0" containerID="60376432365e8d58e0e860343a679a5a16c783b622162cbc7fb58f6f099d10c5" exitCode=60
Oct 30 01:15:02.993770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.993585    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" event={"ID":"e583bfd6-926a-47a6-9885-74675eac92d0","Type":"ContainerDied","Data":"60376432365e8d58e0e860343a679a5a16c783b622162cbc7fb58f6f099d10c5"}
Oct 30 01:15:02.993902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.993781    4576 scope.go:122] "RemoveContainer" containerID="60376432365e8d58e0e860343a679a5a16c783b622162cbc7fb58f6f099d10c5"
Oct 30 01:15:02.995397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.995374    4576 generic.go:356] "Generic (PLEG): container finished" podID="818ea717-759d-4697-9367-6ede754cdc48" containerID="404dbe2cea38e53f8ff35a1776388f3c432fc3d83e4192c7a4ea2467c929b9ad" exitCode=60
Oct 30 01:15:02.995498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.995418    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" event={"ID":"818ea717-759d-4697-9367-6ede754cdc48","Type":"ContainerDied","Data":"404dbe2cea38e53f8ff35a1776388f3c432fc3d83e4192c7a4ea2467c929b9ad"}
Oct 30 01:15:02.995644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.995629    4576 scope.go:122] "RemoveContainer" containerID="404dbe2cea38e53f8ff35a1776388f3c432fc3d83e4192c7a4ea2467c929b9ad"
Oct 30 01:15:02.997154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.997135    4576 generic.go:356] "Generic (PLEG): container finished" podID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerID="7b52b85f3f3299290ec9270803f4d58756afb19ce31607a8c81b0a552035a801" exitCode=60
Oct 30 01:15:02.998124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.997282    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" event={"ID":"e7290ded-d2a3-49d1-b712-dc57bac30f01","Type":"ContainerDied","Data":"7b52b85f3f3299290ec9270803f4d58756afb19ce31607a8c81b0a552035a801"}
Oct 30 01:15:02.998124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.997491    4576 scope.go:122] "RemoveContainer" containerID="7b52b85f3f3299290ec9270803f4d58756afb19ce31607a8c81b0a552035a801"
Oct 30 01:15:02.999111 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.999090    4576 generic.go:356] "Generic (PLEG): container finished" podID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerID="9b4a0c6299fc0c7e46750d9201ee13bcf6ec39e4db942f381436c0b07975f003" exitCode=60
Oct 30 01:15:02.999206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.999155    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" event={"ID":"fff10f99-a41c-46a1-9dc3-5824e57cf03b","Type":"ContainerDied","Data":"9b4a0c6299fc0c7e46750d9201ee13bcf6ec39e4db942f381436c0b07975f003"}
Oct 30 01:15:02.999414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:02.999397    4576 scope.go:122] "RemoveContainer" containerID="9b4a0c6299fc0c7e46750d9201ee13bcf6ec39e4db942f381436c0b07975f003"
Oct 30 01:15:03.003261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:03.001702    4576 generic.go:356] "Generic (PLEG): container finished" podID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerID="d2cdec7f7adcc0e437966dc13a52cdc9e5fd8c6d45f7874b0256ef232e2d7cb3" exitCode=60
Oct 30 01:15:03.003261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:03.001773    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" event={"ID":"58404bec-1cc1-46cc-a5e7-a6994b9c1163","Type":"ContainerDied","Data":"d2cdec7f7adcc0e437966dc13a52cdc9e5fd8c6d45f7874b0256ef232e2d7cb3"}
Oct 30 01:15:03.003261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:03.002403    4576 scope.go:122] "RemoveContainer" containerID="d2cdec7f7adcc0e437966dc13a52cdc9e5fd8c6d45f7874b0256ef232e2d7cb3"
Oct 30 01:15:03.004994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:03.004457    4576 generic.go:356] "Generic (PLEG): container finished" podID="2d98a290-3f4a-4467-ab4d-623a42397144" containerID="fb07cb070a0ef6fd0932b86665c7099b66bb8a55aa693ce9d5d91be31e0ef48c" exitCode=60
Oct 30 01:15:03.004994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:03.004511    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" event={"ID":"2d98a290-3f4a-4467-ab4d-623a42397144","Type":"ContainerDied","Data":"fb07cb070a0ef6fd0932b86665c7099b66bb8a55aa693ce9d5d91be31e0ef48c"}
Oct 30 01:15:03.004994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:03.004741    4576 scope.go:122] "RemoveContainer" containerID="fb07cb070a0ef6fd0932b86665c7099b66bb8a55aa693ce9d5d91be31e0ef48c"
Oct 30 01:15:03.363720 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:03.363676    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:04.015900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.015870    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" event={"ID":"36763be6-dedf-4cc7-b154-430d488e2b33","Type":"ContainerStarted","Data":"7727b0e7070b66c6bd422b78de840a763ad1686e73491435781f8d43baf2800b"}
Oct 30 01:15:04.018175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.018144    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" event={"ID":"5e84f94a-3cd7-48ab-844d-e0605f5a16b2","Type":"ContainerStarted","Data":"276dc2757db6881c18cc58539714107b07fd4e718ecc9cd4d95cc60b2fc25359"}
Oct 30 01:15:04.020275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.020254    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" event={"ID":"497a3e39-1661-457e-8fc4-073c263f2f2c","Type":"ContainerStarted","Data":"7539a4bb4774377cc1841b2dba2627433dadfc2e2a097501a8b5e89bc07df1ff"}
Oct 30 01:15:04.022073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.022053    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" event={"ID":"e583bfd6-926a-47a6-9885-74675eac92d0","Type":"ContainerStarted","Data":"998f3107bbedd05d3aa7a6c1225faac5af5c1d2d8fce88ba55f6c3aa593df3ae"}
Oct 30 01:15:04.023889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.023869    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" event={"ID":"e7290ded-d2a3-49d1-b712-dc57bac30f01","Type":"ContainerStarted","Data":"f4058ce7a63b66fee505e6e0ced855dc3af422b9767648a9dc0b8e77b5e64a8f"}
Oct 30 01:15:04.025745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.025720    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" event={"ID":"58404bec-1cc1-46cc-a5e7-a6994b9c1163","Type":"ContainerStarted","Data":"6b58e6f801a2c08b679fd340700573d00578ef1df7ea870685ff8dfb0af4c9e9"}
Oct 30 01:15:04.027691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.027668    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" event={"ID":"2d98a290-3f4a-4467-ab4d-623a42397144","Type":"ContainerStarted","Data":"f4baf2b1df9bcb77e79e1e3450cd2f98482effcc1558d8fb77c7bd3e1c28c9ed"}
Oct 30 01:15:04.029521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.029499    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" event={"ID":"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85","Type":"ContainerStarted","Data":"45a7151491ed1035214d9212899940c96ffdce41c95aa26681a45a696f192246"}
Oct 30 01:15:04.031353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.031328    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" event={"ID":"54cd525e-0953-419a-9f8b-f9b3c1318918","Type":"ContainerStarted","Data":"573c092fd7c17bbae35aba3a5415cb991c0409d120dd27bb3688be0fd34773ce"}
Oct 30 01:15:04.059488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.059437    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" podStartSLOduration=4.059421494 podStartE2EDuration="4.059421494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.055334305 +0000 UTC m=+2184.126658140" watchObservedRunningTime="2026-10-30 01:15:04.059421494 +0000 UTC m=+2184.130745329"
Oct 30 01:15:04.142129 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.142022    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" podStartSLOduration=4.142008118 podStartE2EDuration="4.142008118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.13951177 +0000 UTC m=+2184.210835620" watchObservedRunningTime="2026-10-30 01:15:04.142008118 +0000 UTC m=+2184.213331954"
Oct 30 01:15:04.181207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.181156    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" podStartSLOduration=4.18114189 podStartE2EDuration="4.18114189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.178817613 +0000 UTC m=+2184.250141448" watchObservedRunningTime="2026-10-30 01:15:04.18114189 +0000 UTC m=+2184.252465724"
Oct 30 01:15:04.223273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.223198    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" podStartSLOduration=4.223183582 podStartE2EDuration="4.223183582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.221893933 +0000 UTC m=+2184.293217768" watchObservedRunningTime="2026-10-30 01:15:04.223183582 +0000 UTC m=+2184.294507417"
Oct 30 01:15:04.351789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.351727    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" podStartSLOduration=4.351706476 podStartE2EDuration="4.351706476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.347755091 +0000 UTC m=+2184.419078926" watchObservedRunningTime="2026-10-30 01:15:04.351706476 +0000 UTC m=+2184.423030313"
Oct 30 01:15:04.388990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.388943    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" podStartSLOduration=4.388927583 podStartE2EDuration="4.388927583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.387946885 +0000 UTC m=+2184.459270721" watchObservedRunningTime="2026-10-30 01:15:04.388927583 +0000 UTC m=+2184.460251417"
Oct 30 01:15:04.718748 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.718720    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"]
Oct 30 01:15:04.723115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.723093    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.739691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.739657    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"]
Oct 30 01:15:04.739953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.739925    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:15:04.829823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.829791    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.830044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.830025    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnbq\" (UniqueName: \"kubernetes.io/projected/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-kube-api-access-2fnbq\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.830087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.830062    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.930946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.930905    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.931152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.931134    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnbq\" (UniqueName: \"kubernetes.io/projected/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-kube-api-access-2fnbq\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.931220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.931180    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.931662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.931634    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.933945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.933927    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:04.952440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:04.952381    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnbq\" (UniqueName: \"kubernetes.io/projected/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-kube-api-access-2fnbq\") pod \"maas-api-revoke-sub-2b97a911-lsklb\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:05.042833 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.042796    4576 generic.go:356] "Generic (PLEG): container finished" podID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerID="276dc2757db6881c18cc58539714107b07fd4e718ecc9cd4d95cc60b2fc25359" exitCode=60
Oct 30 01:15:05.043318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.042865    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" event={"ID":"5e84f94a-3cd7-48ab-844d-e0605f5a16b2","Type":"ContainerDied","Data":"276dc2757db6881c18cc58539714107b07fd4e718ecc9cd4d95cc60b2fc25359"}
Oct 30 01:15:05.043318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.042928    4576 scope.go:122] "RemoveContainer" containerID="85290b4c1f99fba54f3ac1d609c7647ad075d72c765f5d8124c8164f8a2b275a"
Oct 30 01:15:05.043318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.043183    4576 scope.go:122] "RemoveContainer" containerID="276dc2757db6881c18cc58539714107b07fd4e718ecc9cd4d95cc60b2fc25359"
Oct 30 01:15:05.043586 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.043566    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85_odh-ai-gateway-infra(5e84f94a-3cd7-48ab-844d-e0605f5a16b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2"
Oct 30 01:15:05.045054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.045029    4576 generic.go:356] "Generic (PLEG): container finished" podID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerID="7539a4bb4774377cc1841b2dba2627433dadfc2e2a097501a8b5e89bc07df1ff" exitCode=60
Oct 30 01:15:05.045285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.045121    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" event={"ID":"497a3e39-1661-457e-8fc4-073c263f2f2c","Type":"ContainerDied","Data":"7539a4bb4774377cc1841b2dba2627433dadfc2e2a097501a8b5e89bc07df1ff"}
Oct 30 01:15:05.045405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.045381    4576 scope.go:122] "RemoveContainer" containerID="7539a4bb4774377cc1841b2dba2627433dadfc2e2a097501a8b5e89bc07df1ff"
Oct 30 01:15:05.045725 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.045697    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b_odh-ai-gateway-infra(497a3e39-1661-457e-8fc4-073c263f2f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c"
Oct 30 01:15:05.047415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.047386    4576 generic.go:356] "Generic (PLEG): container finished" podID="e583bfd6-926a-47a6-9885-74675eac92d0" containerID="998f3107bbedd05d3aa7a6c1225faac5af5c1d2d8fce88ba55f6c3aa593df3ae" exitCode=60
Oct 30 01:15:05.047504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.047472    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" event={"ID":"e583bfd6-926a-47a6-9885-74675eac92d0","Type":"ContainerDied","Data":"998f3107bbedd05d3aa7a6c1225faac5af5c1d2d8fce88ba55f6c3aa593df3ae"}
Oct 30 01:15:05.047735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.047717    4576 scope.go:122] "RemoveContainer" containerID="998f3107bbedd05d3aa7a6c1225faac5af5c1d2d8fce88ba55f6c3aa593df3ae"
Oct 30 01:15:05.048216 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.048191    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl_odh-ai-gateway-infra(e583bfd6-926a-47a6-9885-74675eac92d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" podUID="e583bfd6-926a-47a6-9885-74675eac92d0"
Oct 30 01:15:05.049641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.049607    4576 generic.go:356] "Generic (PLEG): container finished" podID="818ea717-759d-4697-9367-6ede754cdc48" containerID="db69fec296b84ca20f9d3cf35d93fcc3013501906b9c45fba7f7f123090b81b3" exitCode=60
Oct 30 01:15:05.049730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.049657    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" event={"ID":"818ea717-759d-4697-9367-6ede754cdc48","Type":"ContainerDied","Data":"db69fec296b84ca20f9d3cf35d93fcc3013501906b9c45fba7f7f123090b81b3"}
Oct 30 01:15:05.049917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.049900    4576 scope.go:122] "RemoveContainer" containerID="db69fec296b84ca20f9d3cf35d93fcc3013501906b9c45fba7f7f123090b81b3"
Oct 30 01:15:05.050185 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.050161    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx_odh-ai-gateway-infra(818ea717-759d-4697-9367-6ede754cdc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" podUID="818ea717-759d-4697-9367-6ede754cdc48"
Oct 30 01:15:05.051727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.051705    4576 generic.go:356] "Generic (PLEG): container finished" podID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerID="f4058ce7a63b66fee505e6e0ced855dc3af422b9767648a9dc0b8e77b5e64a8f" exitCode=60
Oct 30 01:15:05.051910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.051892    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" event={"ID":"e7290ded-d2a3-49d1-b712-dc57bac30f01","Type":"ContainerDied","Data":"f4058ce7a63b66fee505e6e0ced855dc3af422b9767648a9dc0b8e77b5e64a8f"}
Oct 30 01:15:05.052013 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.051999    4576 scope.go:122] "RemoveContainer" containerID="f4058ce7a63b66fee505e6e0ced855dc3af422b9767648a9dc0b8e77b5e64a8f"
Oct 30 01:15:05.052267 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.052227    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp_odh-ai-gateway-infra(e7290ded-d2a3-49d1-b712-dc57bac30f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01"
Oct 30 01:15:05.054012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.053933    4576 generic.go:356] "Generic (PLEG): container finished" podID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerID="24835fd1857ebd5db077a481e7dd2243ac098f2778313c8d9966c92cd67e6c93" exitCode=60
Oct 30 01:15:05.054100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.054020    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" event={"ID":"fff10f99-a41c-46a1-9dc3-5824e57cf03b","Type":"ContainerDied","Data":"24835fd1857ebd5db077a481e7dd2243ac098f2778313c8d9966c92cd67e6c93"}
Oct 30 01:15:05.054311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.054292    4576 scope.go:122] "RemoveContainer" containerID="24835fd1857ebd5db077a481e7dd2243ac098f2778313c8d9966c92cd67e6c93"
Oct 30 01:15:05.054579 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.054544    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww_odh-ai-gateway-infra(fff10f99-a41c-46a1-9dc3-5824e57cf03b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b"
Oct 30 01:15:05.056154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.056132    4576 generic.go:356] "Generic (PLEG): container finished" podID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerID="6b58e6f801a2c08b679fd340700573d00578ef1df7ea870685ff8dfb0af4c9e9" exitCode=60
Oct 30 01:15:05.056243 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.056209    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" event={"ID":"58404bec-1cc1-46cc-a5e7-a6994b9c1163","Type":"ContainerDied","Data":"6b58e6f801a2c08b679fd340700573d00578ef1df7ea870685ff8dfb0af4c9e9"}
Oct 30 01:15:05.056407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.056391    4576 scope.go:122] "RemoveContainer" containerID="6b58e6f801a2c08b679fd340700573d00578ef1df7ea870685ff8dfb0af4c9e9"
Oct 30 01:15:05.056622 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.056605    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v_odh-ai-gateway-infra(58404bec-1cc1-46cc-a5e7-a6994b9c1163)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163"
Oct 30 01:15:05.058041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.058022    4576 generic.go:356] "Generic (PLEG): container finished" podID="2d98a290-3f4a-4467-ab4d-623a42397144" containerID="f4baf2b1df9bcb77e79e1e3450cd2f98482effcc1558d8fb77c7bd3e1c28c9ed" exitCode=60
Oct 30 01:15:05.058171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.058048    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" event={"ID":"2d98a290-3f4a-4467-ab4d-623a42397144","Type":"ContainerDied","Data":"f4baf2b1df9bcb77e79e1e3450cd2f98482effcc1558d8fb77c7bd3e1c28c9ed"}
Oct 30 01:15:05.058331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.058315    4576 scope.go:122] "RemoveContainer" containerID="f4baf2b1df9bcb77e79e1e3450cd2f98482effcc1558d8fb77c7bd3e1c28c9ed"
Oct 30 01:15:05.058559 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.058534    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm_odh-ai-gateway-infra(2d98a290-3f4a-4467-ab4d-623a42397144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" podUID="2d98a290-3f4a-4467-ab4d-623a42397144"
Oct 30 01:15:05.058776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.058759    4576 scope.go:122] "RemoveContainer" containerID="d3326e2526998cce24946e263f31182b89c76d8c5e8a43e56340f9a651e940ab"
Oct 30 01:15:05.060173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.060151    4576 generic.go:356] "Generic (PLEG): container finished" podID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerID="45a7151491ed1035214d9212899940c96ffdce41c95aa26681a45a696f192246" exitCode=60
Oct 30 01:15:05.060326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.060253    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" event={"ID":"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85","Type":"ContainerDied","Data":"45a7151491ed1035214d9212899940c96ffdce41c95aa26681a45a696f192246"}
Oct 30 01:15:05.060450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.060438    4576 scope.go:122] "RemoveContainer" containerID="45a7151491ed1035214d9212899940c96ffdce41c95aa26681a45a696f192246"
Oct 30 01:15:05.060648 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.060631    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l_odh-ai-gateway-infra(5f3cfe9b-07b2-4a74-a04d-40a8f55efa85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85"
Oct 30 01:15:05.062282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.062219    4576 generic.go:356] "Generic (PLEG): container finished" podID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerID="573c092fd7c17bbae35aba3a5415cb991c0409d120dd27bb3688be0fd34773ce" exitCode=60
Oct 30 01:15:05.062282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.062272    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" event={"ID":"54cd525e-0953-419a-9f8b-f9b3c1318918","Type":"ContainerDied","Data":"573c092fd7c17bbae35aba3a5415cb991c0409d120dd27bb3688be0fd34773ce"}
Oct 30 01:15:05.062507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.062491    4576 scope.go:122] "RemoveContainer" containerID="573c092fd7c17bbae35aba3a5415cb991c0409d120dd27bb3688be0fd34773ce"
Oct 30 01:15:05.062701 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.062680    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8_odh-ai-gateway-infra(54cd525e-0953-419a-9f8b-f9b3c1318918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918"
Oct 30 01:15:05.064157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.064137    4576 generic.go:356] "Generic (PLEG): container finished" podID="36763be6-dedf-4cc7-b154-430d488e2b33" containerID="7727b0e7070b66c6bd422b78de840a763ad1686e73491435781f8d43baf2800b" exitCode=60
Oct 30 01:15:05.064299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.064211    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" event={"ID":"36763be6-dedf-4cc7-b154-430d488e2b33","Type":"ContainerDied","Data":"7727b0e7070b66c6bd422b78de840a763ad1686e73491435781f8d43baf2800b"}
Oct 30 01:15:05.064393 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.064379    4576 scope.go:122] "RemoveContainer" containerID="7727b0e7070b66c6bd422b78de840a763ad1686e73491435781f8d43baf2800b"
Oct 30 01:15:05.064570 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:05.064552    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf_odh-ai-gateway-infra(36763be6-dedf-4cc7-b154-430d488e2b33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" podUID="36763be6-dedf-4cc7-b154-430d488e2b33"
Oct 30 01:15:05.076601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.076578    4576 scope.go:122] "RemoveContainer" containerID="60376432365e8d58e0e860343a679a5a16c783b622162cbc7fb58f6f099d10c5"
Oct 30 01:15:05.094376 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.094359    4576 scope.go:122] "RemoveContainer" containerID="404dbe2cea38e53f8ff35a1776388f3c432fc3d83e4192c7a4ea2467c929b9ad"
Oct 30 01:15:05.109113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.109096    4576 scope.go:122] "RemoveContainer" containerID="7b52b85f3f3299290ec9270803f4d58756afb19ce31607a8c81b0a552035a801"
Oct 30 01:15:05.123991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.123971    4576 scope.go:122] "RemoveContainer" containerID="9b4a0c6299fc0c7e46750d9201ee13bcf6ec39e4db942f381436c0b07975f003"
Oct 30 01:15:05.138952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.138934    4576 scope.go:122] "RemoveContainer" containerID="d2cdec7f7adcc0e437966dc13a52cdc9e5fd8c6d45f7874b0256ef232e2d7cb3"
Oct 30 01:15:05.145245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.145209    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:05.157827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.157796    4576 scope.go:122] "RemoveContainer" containerID="fb07cb070a0ef6fd0932b86665c7099b66bb8a55aa693ce9d5d91be31e0ef48c"
Oct 30 01:15:05.172924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.172898    4576 scope.go:122] "RemoveContainer" containerID="ef01da02e1e539e0f3e99d7e3ae2584d618e90f80c3b904d37ab6415757dbf70"
Oct 30 01:15:05.189523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.189494    4576 scope.go:122] "RemoveContainer" containerID="a777989a1180eb98dde4184984d5ab8dbba47a6c8d2535e4cf9a3e9e23af5f7b"
Oct 30 01:15:05.205631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.205507    4576 scope.go:122] "RemoveContainer" containerID="b6fb0b68fa410850cfbed70f46078f3e273784a8c4c5b70e80ec3f4c039453c0"
Oct 30 01:15:05.276435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.276388    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:05.460274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:05.460204    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"]
Oct 30 01:15:05.461138 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:05.461107    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e86a82_5c14_4287_a9d9_05c7b4e7d1b2.slice/crio-97e6f1fe529ecb1a62584bc49f7fd978d50d1a57492c47904d25d6320f01d305 WatchSource:0}: Error finding container 97e6f1fe529ecb1a62584bc49f7fd978d50d1a57492c47904d25d6320f01d305: Status 404 returned error can't find the container with id 97e6f1fe529ecb1a62584bc49f7fd978d50d1a57492c47904d25d6320f01d305
Oct 30 01:15:06.074827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.074751    4576 scope.go:122] "RemoveContainer" containerID="6b58e6f801a2c08b679fd340700573d00578ef1df7ea870685ff8dfb0af4c9e9"
Oct 30 01:15:06.075278 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.075094    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v_odh-ai-gateway-infra(58404bec-1cc1-46cc-a5e7-a6994b9c1163)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163"
Oct 30 01:15:06.076567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.076545    4576 scope.go:122] "RemoveContainer" containerID="45a7151491ed1035214d9212899940c96ffdce41c95aa26681a45a696f192246"
Oct 30 01:15:06.076884 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.076863    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l_odh-ai-gateway-infra(5f3cfe9b-07b2-4a74-a04d-40a8f55efa85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85"
Oct 30 01:15:06.078276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.078256    4576 scope.go:122] "RemoveContainer" containerID="573c092fd7c17bbae35aba3a5415cb991c0409d120dd27bb3688be0fd34773ce"
Oct 30 01:15:06.078563 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.078538    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8_odh-ai-gateway-infra(54cd525e-0953-419a-9f8b-f9b3c1318918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918"
Oct 30 01:15:06.079898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.079877    4576 scope.go:122] "RemoveContainer" containerID="276dc2757db6881c18cc58539714107b07fd4e718ecc9cd4d95cc60b2fc25359"
Oct 30 01:15:06.080102 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.080078    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85_odh-ai-gateway-infra(5e84f94a-3cd7-48ab-844d-e0605f5a16b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2"
Oct 30 01:15:06.081612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.081592    4576 scope.go:122] "RemoveContainer" containerID="7539a4bb4774377cc1841b2dba2627433dadfc2e2a097501a8b5e89bc07df1ff"
Oct 30 01:15:06.081842 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.081819    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b_odh-ai-gateway-infra(497a3e39-1661-457e-8fc4-073c263f2f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c"
Oct 30 01:15:06.083202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.083182    4576 scope.go:122] "RemoveContainer" containerID="db69fec296b84ca20f9d3cf35d93fcc3013501906b9c45fba7f7f123090b81b3"
Oct 30 01:15:06.083447 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.083422    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx_odh-ai-gateway-infra(818ea717-759d-4697-9367-6ede754cdc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" podUID="818ea717-759d-4697-9367-6ede754cdc48"
Oct 30 01:15:06.084824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.084804    4576 scope.go:122] "RemoveContainer" containerID="24835fd1857ebd5db077a481e7dd2243ac098f2778313c8d9966c92cd67e6c93"
Oct 30 01:15:06.085037 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.085012    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww_odh-ai-gateway-infra(fff10f99-a41c-46a1-9dc3-5824e57cf03b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b"
Oct 30 01:15:06.087601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.087582    4576 scope.go:122] "RemoveContainer" containerID="f4baf2b1df9bcb77e79e1e3450cd2f98482effcc1558d8fb77c7bd3e1c28c9ed"
Oct 30 01:15:06.087798 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.087781    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm_odh-ai-gateway-infra(2d98a290-3f4a-4467-ab4d-623a42397144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" podUID="2d98a290-3f4a-4467-ab4d-623a42397144"
Oct 30 01:15:06.089197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.089180    4576 scope.go:122] "RemoveContainer" containerID="7727b0e7070b66c6bd422b78de840a763ad1686e73491435781f8d43baf2800b"
Oct 30 01:15:06.089436 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.089414    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf_odh-ai-gateway-infra(36763be6-dedf-4cc7-b154-430d488e2b33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" podUID="36763be6-dedf-4cc7-b154-430d488e2b33"
Oct 30 01:15:06.090665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.090640    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb" event={"ID":"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2","Type":"ContainerStarted","Data":"7cad2c55e89c2ff50bb1df84275aa6c93701df7b13f45506d6d72b9ff59996dd"}
Oct 30 01:15:06.090763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.090688    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb" event={"ID":"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2","Type":"ContainerStarted","Data":"97e6f1fe529ecb1a62584bc49f7fd978d50d1a57492c47904d25d6320f01d305"}
Oct 30 01:15:06.092566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.092547    4576 scope.go:122] "RemoveContainer" containerID="998f3107bbedd05d3aa7a6c1225faac5af5c1d2d8fce88ba55f6c3aa593df3ae"
Oct 30 01:15:06.092781 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.092757    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl_odh-ai-gateway-infra(e583bfd6-926a-47a6-9885-74675eac92d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" podUID="e583bfd6-926a-47a6-9885-74675eac92d0"
Oct 30 01:15:06.094118 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.094102    4576 scope.go:122] "RemoveContainer" containerID="f4058ce7a63b66fee505e6e0ced855dc3af422b9767648a9dc0b8e77b5e64a8f"
Oct 30 01:15:06.094320 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:06.094303    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp_odh-ai-gateway-infra(e7290ded-d2a3-49d1-b712-dc57bac30f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01"
Oct 30 01:15:06.161909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.161851    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb" podStartSLOduration=2.161832968 podStartE2EDuration="2.161832968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.160963407 +0000 UTC m=+2186.232287243" watchObservedRunningTime="2026-10-30 01:15:06.161832968 +0000 UTC m=+2186.233156803"
Oct 30 01:15:06.899754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:06.899708    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:07.103427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:07.103396    4576 generic.go:356] "Generic (PLEG): container finished" podID="e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2" containerID="7cad2c55e89c2ff50bb1df84275aa6c93701df7b13f45506d6d72b9ff59996dd" exitCode=0
Oct 30 01:15:07.103810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:07.103482    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb" event={"ID":"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2","Type":"ContainerDied","Data":"7cad2c55e89c2ff50bb1df84275aa6c93701df7b13f45506d6d72b9ff59996dd"}
Oct 30 01:15:08.277322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.277297    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:08.382271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.382153    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-kube-api-access-2fnbq\" (UniqueName: \"kubernetes.io/projected/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-kube-api-access-2fnbq\") pod \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") "
Oct 30 01:15:08.382271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.382220    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-openshift-ca\") pod \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") "
Oct 30 01:15:08.382510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.382387    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-xks-ca\") pod \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\" (UID: \"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2\") "
Oct 30 01:15:08.382567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.382546    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-openshift-ca" pod "e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2" (UID: "e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:08.382965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.382943    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:08.384505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.384483    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-kube-api-access-2fnbq" pod "e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2" (UID: "e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2"). InnerVolumeSpecName "kube-api-access-2fnbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:08.384650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.384631    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-xks-ca" pod "e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2" (UID: "e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:08.439617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.439570    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:08.484577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.484545    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fnbq\" (UniqueName: \"kubernetes.io/projected/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-kube-api-access-2fnbq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:08.484577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:08.484573    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:09.120921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:09.120890    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"
Oct 30 01:15:09.121102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:09.120889    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb" event={"ID":"e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2","Type":"ContainerDied","Data":"97e6f1fe529ecb1a62584bc49f7fd978d50d1a57492c47904d25d6320f01d305"}
Oct 30 01:15:09.121102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:09.121010    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e6f1fe529ecb1a62584bc49f7fd978d50d1a57492c47904d25d6320f01d305"
Oct 30 01:15:10.065530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:10.065482    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:11.587418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:11.587380    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:13.059381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:13.059341    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:14.679344 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:14.679295    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:15.102283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.102252    4576 scope.go:122] "RemoveContainer" containerID="7539a4bb4774377cc1841b2dba2627433dadfc2e2a097501a8b5e89bc07df1ff"
Oct 30 01:15:15.213854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.213818    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"]
Oct 30 01:15:15.214574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.214555    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2" containerName="revoke-keys"
Oct 30 01:15:15.214574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.214575    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2" containerName="revoke-keys"
Oct 30 01:15:15.214704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.214693    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2" containerName="revoke-keys"
Oct 30 01:15:15.219735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.219704    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.235600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.235571    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"]
Oct 30 01:15:15.245313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.245278    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-d0ae0d\"/\"default-dockercfg-cv4f2\""
Oct 30 01:15:15.245461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.245315    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-d0ae0d\"/\"kube-root-ca.crt\""
Oct 30 01:15:15.249895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.249871    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-d0ae0d\"/\"openshift-service-ca.crt\""
Oct 30 01:15:15.250030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.249981    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-d0ae0d\"/\"rate-test-model-d0ae0d-kserve-self-signed-certs\""
Oct 30 01:15:15.281975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.281541    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:15.384956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.384924    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprml\" (UniqueName: \"kubernetes.io/projected/eb2e883e-babe-4ffd-bafb-369d08bc9b12-kube-api-access-fprml\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.385113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.385090    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-model-cache\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.385211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.385193    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tls-certs\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.385404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.385377    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-home\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.385456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.385437    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-dshm\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.385514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.385502    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tmp-dir\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.487256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.486738    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-home\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.487256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.486882    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-dshm\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.487256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.486940    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tmp-dir\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.487256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.486975    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprml\" (UniqueName: \"kubernetes.io/projected/eb2e883e-babe-4ffd-bafb-369d08bc9b12-kube-api-access-fprml\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.487256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.487073    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-model-cache\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.489775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.489747    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tmp-dir\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.494255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.490323    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tls-certs\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.494255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.490370    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-home\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.494255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.490711    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-model-cache\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.499253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.496138    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tls-certs\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.504542 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.504508    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-dshm\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.518141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.516396    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprml\" (UniqueName: \"kubernetes.io/projected/eb2e883e-babe-4ffd-bafb-369d08bc9b12-kube-api-access-fprml\") pod \"rate-test-model-d0ae0d-kserve-856fb476f-hcwc9\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") " pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.537411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.537367    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:15.855506 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:15.855471    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb2e883e_babe_4ffd_bafb_369d08bc9b12.slice/crio-d1a220bc4986564266046d76ee3c8c78f5f7134aad86f7f5986e66eeab10fa8e WatchSource:0}: Error finding container d1a220bc4986564266046d76ee3c8c78f5f7134aad86f7f5986e66eeab10fa8e: Status 404 returned error can't find the container with id d1a220bc4986564266046d76ee3c8c78f5f7134aad86f7f5986e66eeab10fa8e
Oct 30 01:15:15.855940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:15.855591    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"]
Oct 30 01:15:16.101857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.101779    4576 scope.go:122] "RemoveContainer" containerID="6b58e6f801a2c08b679fd340700573d00578ef1df7ea870685ff8dfb0af4c9e9"
Oct 30 01:15:16.102024 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.101902    4576 scope.go:122] "RemoveContainer" containerID="f4baf2b1df9bcb77e79e1e3450cd2f98482effcc1558d8fb77c7bd3e1c28c9ed"
Oct 30 01:15:16.102097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.102028    4576 scope.go:122] "RemoveContainer" containerID="24835fd1857ebd5db077a481e7dd2243ac098f2778313c8d9966c92cd67e6c93"
Oct 30 01:15:16.191640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.191610    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9" event={"ID":"eb2e883e-babe-4ffd-bafb-369d08bc9b12","Type":"ContainerStarted","Data":"56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac"}
Oct 30 01:15:16.191768 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.191659    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9" event={"ID":"eb2e883e-babe-4ffd-bafb-369d08bc9b12","Type":"ContainerStarted","Data":"d1a220bc4986564266046d76ee3c8c78f5f7134aad86f7f5986e66eeab10fa8e"}
Oct 30 01:15:16.193541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.193515    4576 generic.go:356] "Generic (PLEG): container finished" podID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerID="007df25afb28577259b4957efc2bce58a8ea41865f1e887fef9f47761cc322d9" exitCode=60
Oct 30 01:15:16.193664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.193650    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" event={"ID":"497a3e39-1661-457e-8fc4-073c263f2f2c","Type":"ContainerDied","Data":"007df25afb28577259b4957efc2bce58a8ea41865f1e887fef9f47761cc322d9"}
Oct 30 01:15:16.193717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.193693    4576 scope.go:122] "RemoveContainer" containerID="7539a4bb4774377cc1841b2dba2627433dadfc2e2a097501a8b5e89bc07df1ff"
Oct 30 01:15:16.194067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.194053    4576 scope.go:122] "RemoveContainer" containerID="007df25afb28577259b4957efc2bce58a8ea41865f1e887fef9f47761cc322d9"
Oct 30 01:15:16.194439 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:16.194419    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b_odh-ai-gateway-infra(497a3e39-1661-457e-8fc4-073c263f2f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c"
Oct 30 01:15:16.235947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.235893    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9" podStartSLOduration=1.235877766 podStartE2EDuration="1.235877766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:16.231977813 +0000 UTC m=+2196.303301649" watchObservedRunningTime="2026-10-30 01:15:16.235877766 +0000 UTC m=+2196.307201602"
Oct 30 01:15:16.348632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.348601    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:16.537732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:16.537700    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:17.101411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.101376    4576 scope.go:122] "RemoveContainer" containerID="573c092fd7c17bbae35aba3a5415cb991c0409d120dd27bb3688be0fd34773ce"
Oct 30 01:15:17.101888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.101583    4576 scope.go:122] "RemoveContainer" containerID="45a7151491ed1035214d9212899940c96ffdce41c95aa26681a45a696f192246"
Oct 30 01:15:17.101888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.101812    4576 scope.go:122] "RemoveContainer" containerID="276dc2757db6881c18cc58539714107b07fd4e718ecc9cd4d95cc60b2fc25359"
Oct 30 01:15:17.101982 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.101971    4576 scope.go:122] "RemoveContainer" containerID="7727b0e7070b66c6bd422b78de840a763ad1686e73491435781f8d43baf2800b"
Oct 30 01:15:17.203866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.203826    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" event={"ID":"58404bec-1cc1-46cc-a5e7-a6994b9c1163","Type":"ContainerStarted","Data":"c986649431413d911789e270854772718c6b9c11ed2362c9b0a561166e4fd310"}
Oct 30 01:15:17.207966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.207918    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" event={"ID":"fff10f99-a41c-46a1-9dc3-5824e57cf03b","Type":"ContainerStarted","Data":"5b7209e1a82bc8c592b4e15e0153d877075e3aa3efd52f86bc0eebd1e00a87a5"}
Oct 30 01:15:17.210312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.210283    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" event={"ID":"2d98a290-3f4a-4467-ab4d-623a42397144","Type":"ContainerStarted","Data":"2465222888add22f01cb19deeb55b76bf16a3ef30f3459c08a95f7f586724e3c"}
Oct 30 01:15:17.285738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.285705    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"]
Oct 30 01:15:17.586664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.586638    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:17.734465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.734435    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497a3e39-1661-457e-8fc4-073c263f2f2c-kube-api-access-d8cxh\" (UniqueName: \"kubernetes.io/projected/497a3e39-1661-457e-8fc4-073c263f2f2c-kube-api-access-d8cxh\") pod \"497a3e39-1661-457e-8fc4-073c263f2f2c\" (UID: \"497a3e39-1661-457e-8fc4-073c263f2f2c\") "
Oct 30 01:15:17.734641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.734574    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497a3e39-1661-457e-8fc4-073c263f2f2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/497a3e39-1661-457e-8fc4-073c263f2f2c-maas-api-service-ca\") pod \"497a3e39-1661-457e-8fc4-073c263f2f2c\" (UID: \"497a3e39-1661-457e-8fc4-073c263f2f2c\") "
Oct 30 01:15:17.734888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.734861    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497a3e39-1661-457e-8fc4-073c263f2f2c-maas-api-service-ca" pod "497a3e39-1661-457e-8fc4-073c263f2f2c" (UID: "497a3e39-1661-457e-8fc4-073c263f2f2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:17.735117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.735103    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/497a3e39-1661-457e-8fc4-073c263f2f2c-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:17.768252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.768199    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497a3e39-1661-457e-8fc4-073c263f2f2c-kube-api-access-d8cxh" pod "497a3e39-1661-457e-8fc4-073c263f2f2c" (UID: "497a3e39-1661-457e-8fc4-073c263f2f2c"). InnerVolumeSpecName "kube-api-access-d8cxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:17.836802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.836767    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8cxh\" (UniqueName: \"kubernetes.io/projected/497a3e39-1661-457e-8fc4-073c263f2f2c-kube-api-access-d8cxh\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:17.940954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:17.940915    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:18.051509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.051472    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:18.102137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.102099    4576 scope.go:122] "RemoveContainer" containerID="f4058ce7a63b66fee505e6e0ced855dc3af422b9767648a9dc0b8e77b5e64a8f"
Oct 30 01:15:18.102938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.102911    4576 scope.go:122] "RemoveContainer" containerID="db69fec296b84ca20f9d3cf35d93fcc3013501906b9c45fba7f7f123090b81b3"
Oct 30 01:15:18.172270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.171024    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:18.222442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.221252    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" event={"ID":"5e84f94a-3cd7-48ab-844d-e0605f5a16b2","Type":"ContainerStarted","Data":"e8f60380206047e6e8880159597cb3b8a8a9935b935ab808cf8f479dafb3c5ef"}
Oct 30 01:15:18.222722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.222704    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"
Oct 30 01:15:18.222800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.222772    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b" event={"ID":"497a3e39-1661-457e-8fc4-073c263f2f2c","Type":"ContainerDied","Data":"f3d824914a6e0a3eede7325fdef84969744b7971f7d3345c3cae2729b2036d5d"}
Oct 30 01:15:18.222903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.222834    4576 scope.go:122] "RemoveContainer" containerID="007df25afb28577259b4957efc2bce58a8ea41865f1e887fef9f47761cc322d9"
Oct 30 01:15:18.225718 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.225690    4576 generic.go:356] "Generic (PLEG): container finished" podID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerID="5b7209e1a82bc8c592b4e15e0153d877075e3aa3efd52f86bc0eebd1e00a87a5" exitCode=60
Oct 30 01:15:18.225799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.225752    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" event={"ID":"fff10f99-a41c-46a1-9dc3-5824e57cf03b","Type":"ContainerDied","Data":"5b7209e1a82bc8c592b4e15e0153d877075e3aa3efd52f86bc0eebd1e00a87a5"}
Oct 30 01:15:18.226135 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.226119    4576 scope.go:122] "RemoveContainer" containerID="5b7209e1a82bc8c592b4e15e0153d877075e3aa3efd52f86bc0eebd1e00a87a5"
Oct 30 01:15:18.226484 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:18.226464    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww_odh-ai-gateway-infra(fff10f99-a41c-46a1-9dc3-5824e57cf03b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b"
Oct 30 01:15:18.227867 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.227843    4576 generic.go:356] "Generic (PLEG): container finished" podID="2d98a290-3f4a-4467-ab4d-623a42397144" containerID="2465222888add22f01cb19deeb55b76bf16a3ef30f3459c08a95f7f586724e3c" exitCode=60
Oct 30 01:15:18.227961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.227917    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" event={"ID":"2d98a290-3f4a-4467-ab4d-623a42397144","Type":"ContainerDied","Data":"2465222888add22f01cb19deeb55b76bf16a3ef30f3459c08a95f7f586724e3c"}
Oct 30 01:15:18.228315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.228294    4576 scope.go:122] "RemoveContainer" containerID="2465222888add22f01cb19deeb55b76bf16a3ef30f3459c08a95f7f586724e3c"
Oct 30 01:15:18.228624 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:18.228600    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm_odh-ai-gateway-infra(2d98a290-3f4a-4467-ab4d-623a42397144)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" podUID="2d98a290-3f4a-4467-ab4d-623a42397144"
Oct 30 01:15:18.229991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.229973    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" event={"ID":"36763be6-dedf-4cc7-b154-430d488e2b33","Type":"ContainerStarted","Data":"57dab9bbf3ce768b426012064276affa0ca30d33b885bb7aa1e6272a34b38b97"}
Oct 30 01:15:18.231918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.231897    4576 generic.go:356] "Generic (PLEG): container finished" podID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerID="c986649431413d911789e270854772718c6b9c11ed2362c9b0a561166e4fd310" exitCode=60
Oct 30 01:15:18.232019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.231942    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" event={"ID":"58404bec-1cc1-46cc-a5e7-a6994b9c1163","Type":"ContainerDied","Data":"c986649431413d911789e270854772718c6b9c11ed2362c9b0a561166e4fd310"}
Oct 30 01:15:18.232275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.232257    4576 scope.go:122] "RemoveContainer" containerID="c986649431413d911789e270854772718c6b9c11ed2362c9b0a561166e4fd310"
Oct 30 01:15:18.232543 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:18.232525    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v_odh-ai-gateway-infra(58404bec-1cc1-46cc-a5e7-a6994b9c1163)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163"
Oct 30 01:15:18.234054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.234029    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" event={"ID":"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85","Type":"ContainerStarted","Data":"1b5d5b57bcd0552ad3b0974aaaf942e229eed6eae6d85023622b6960d7e38a5e"}
Oct 30 01:15:18.234724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.234703    4576 scope.go:122] "RemoveContainer" containerID="24835fd1857ebd5db077a481e7dd2243ac098f2778313c8d9966c92cd67e6c93"
Oct 30 01:15:18.236146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.236128    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" event={"ID":"54cd525e-0953-419a-9f8b-f9b3c1318918","Type":"ContainerStarted","Data":"f307edade830cb5f39696a7527e765cdce8b99e519a7df62da87d747a8490bd8"}
Oct 30 01:15:18.249537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.249515    4576 scope.go:122] "RemoveContainer" containerID="f4baf2b1df9bcb77e79e1e3450cd2f98482effcc1558d8fb77c7bd3e1c28c9ed"
Oct 30 01:15:18.255129 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.255104    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"]
Oct 30 01:15:18.264344 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.264225    4576 scope.go:122] "RemoveContainer" containerID="6b58e6f801a2c08b679fd340700573d00578ef1df7ea870685ff8dfb0af4c9e9"
Oct 30 01:15:18.296477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.296451    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"]
Oct 30 01:15:18.337378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.337345    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"]
Oct 30 01:15:18.398329 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:18.398186    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:15:18.468648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.468539    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"]
Oct 30 01:15:18.476757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:18.476731    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8cf33a-29888715-f5m4b"]
Oct 30 01:15:19.101967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.101889    4576 scope.go:122] "RemoveContainer" containerID="998f3107bbedd05d3aa7a6c1225faac5af5c1d2d8fce88ba55f6c3aa593df3ae"
Oct 30 01:15:19.252922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.252892    4576 generic.go:356] "Generic (PLEG): container finished" podID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerID="1b5d5b57bcd0552ad3b0974aaaf942e229eed6eae6d85023622b6960d7e38a5e" exitCode=60
Oct 30 01:15:19.253395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.252945    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" event={"ID":"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85","Type":"ContainerDied","Data":"1b5d5b57bcd0552ad3b0974aaaf942e229eed6eae6d85023622b6960d7e38a5e"}
Oct 30 01:15:19.253395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.253005    4576 scope.go:122] "RemoveContainer" containerID="45a7151491ed1035214d9212899940c96ffdce41c95aa26681a45a696f192246"
Oct 30 01:15:19.253534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.253475    4576 scope.go:122] "RemoveContainer" containerID="1b5d5b57bcd0552ad3b0974aaaf942e229eed6eae6d85023622b6960d7e38a5e"
Oct 30 01:15:19.254596 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:19.253826    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l_odh-ai-gateway-infra(5f3cfe9b-07b2-4a74-a04d-40a8f55efa85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85"
Oct 30 01:15:19.256169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.256145    4576 generic.go:356] "Generic (PLEG): container finished" podID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerID="f307edade830cb5f39696a7527e765cdce8b99e519a7df62da87d747a8490bd8" exitCode=60
Oct 30 01:15:19.256301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.256214    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" event={"ID":"54cd525e-0953-419a-9f8b-f9b3c1318918","Type":"ContainerDied","Data":"f307edade830cb5f39696a7527e765cdce8b99e519a7df62da87d747a8490bd8"}
Oct 30 01:15:19.256652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.256633    4576 scope.go:122] "RemoveContainer" containerID="f307edade830cb5f39696a7527e765cdce8b99e519a7df62da87d747a8490bd8"
Oct 30 01:15:19.257015 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:19.256986    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8_odh-ai-gateway-infra(54cd525e-0953-419a-9f8b-f9b3c1318918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918"
Oct 30 01:15:19.258288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.258227    4576 generic.go:356] "Generic (PLEG): container finished" podID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerID="e8f60380206047e6e8880159597cb3b8a8a9935b935ab808cf8f479dafb3c5ef" exitCode=60
Oct 30 01:15:19.258372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.258324    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" event={"ID":"5e84f94a-3cd7-48ab-844d-e0605f5a16b2","Type":"ContainerDied","Data":"e8f60380206047e6e8880159597cb3b8a8a9935b935ab808cf8f479dafb3c5ef"}
Oct 30 01:15:19.258751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.258731    4576 scope.go:122] "RemoveContainer" containerID="e8f60380206047e6e8880159597cb3b8a8a9935b935ab808cf8f479dafb3c5ef"
Oct 30 01:15:19.259161 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:19.259117    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85_odh-ai-gateway-infra(5e84f94a-3cd7-48ab-844d-e0605f5a16b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2"
Oct 30 01:15:19.261985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.261958    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" event={"ID":"818ea717-759d-4697-9367-6ede754cdc48","Type":"ContainerStarted","Data":"2ef831b8c4cbf50c4d64f2e6ee425fd4dbfa9dc91152b0437481fa353e1c0fa2"}
Oct 30 01:15:19.272382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.272199    4576 scope.go:122] "RemoveContainer" containerID="573c092fd7c17bbae35aba3a5415cb991c0409d120dd27bb3688be0fd34773ce"
Oct 30 01:15:19.288431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.288349    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"]
Oct 30 01:15:19.289095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.288647    4576 generic.go:356] "Generic (PLEG): container finished" podID="36763be6-dedf-4cc7-b154-430d488e2b33" containerID="57dab9bbf3ce768b426012064276affa0ca30d33b885bb7aa1e6272a34b38b97" exitCode=60
Oct 30 01:15:19.289095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.288856    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" event={"ID":"36763be6-dedf-4cc7-b154-430d488e2b33","Type":"ContainerDied","Data":"57dab9bbf3ce768b426012064276affa0ca30d33b885bb7aa1e6272a34b38b97"}
Oct 30 01:15:19.289643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.289441    4576 scope.go:122] "RemoveContainer" containerID="57dab9bbf3ce768b426012064276affa0ca30d33b885bb7aa1e6272a34b38b97"
Oct 30 01:15:19.291154 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:19.290040    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf_odh-ai-gateway-infra(36763be6-dedf-4cc7-b154-430d488e2b33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" podUID="36763be6-dedf-4cc7-b154-430d488e2b33"
Oct 30 01:15:19.310310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.310279    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" event={"ID":"e7290ded-d2a3-49d1-b712-dc57bac30f01","Type":"ContainerStarted","Data":"67acc3f21e0d96e09422b6247cdf945039a2f1415e3c673d1a1618b4c5a93e51"}
Oct 30 01:15:19.317923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.317895    4576 scope.go:122] "RemoveContainer" containerID="276dc2757db6881c18cc58539714107b07fd4e718ecc9cd4d95cc60b2fc25359"
Oct 30 01:15:19.326371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.326340    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"]
Oct 30 01:15:19.337977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.337952    4576 scope.go:122] "RemoveContainer" containerID="7727b0e7070b66c6bd422b78de840a763ad1686e73491435781f8d43baf2800b"
Oct 30 01:15:19.387497 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.387163    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"]
Oct 30 01:15:19.460867 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.460837    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"]
Oct 30 01:15:19.530463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.530433    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:19.596595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.596541    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" podStartSLOduration=19.596518357 podStartE2EDuration="19.596518357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:19.595960459 +0000 UTC m=+2199.667284293" watchObservedRunningTime="2026-10-30 01:15:19.596518357 +0000 UTC m=+2199.667842192"
Oct 30 01:15:19.610727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.610701    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58404bec-1cc1-46cc-a5e7-a6994b9c1163-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58404bec-1cc1-46cc-a5e7-a6994b9c1163-maas-api-service-ca\") pod \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\" (UID: \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\") "
Oct 30 01:15:19.611043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.611023    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58404bec-1cc1-46cc-a5e7-a6994b9c1163-maas-api-service-ca" pod "58404bec-1cc1-46cc-a5e7-a6994b9c1163" (UID: "58404bec-1cc1-46cc-a5e7-a6994b9c1163"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:19.611201 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.611183    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58404bec-1cc1-46cc-a5e7-a6994b9c1163-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:19.615935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.615901    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:19.619108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.619050    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:19.712034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.711998    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d98a290-3f4a-4467-ab4d-623a42397144-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98a290-3f4a-4467-ab4d-623a42397144-maas-api-service-ca\") pod \"2d98a290-3f4a-4467-ab4d-623a42397144\" (UID: \"2d98a290-3f4a-4467-ab4d-623a42397144\") "
Oct 30 01:15:19.712225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.712106    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff10f99-a41c-46a1-9dc3-5824e57cf03b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff10f99-a41c-46a1-9dc3-5824e57cf03b-maas-api-service-ca\") pod \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\" (UID: \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\") "
Oct 30 01:15:19.712225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.712214    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff10f99-a41c-46a1-9dc3-5824e57cf03b-kube-api-access-vvbxs\" (UniqueName: \"kubernetes.io/projected/fff10f99-a41c-46a1-9dc3-5824e57cf03b-kube-api-access-vvbxs\") pod \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\" (UID: \"fff10f99-a41c-46a1-9dc3-5824e57cf03b\") "
Oct 30 01:15:19.712405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.712317    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d98a290-3f4a-4467-ab4d-623a42397144-kube-api-access-6vdtz\" (UniqueName: \"kubernetes.io/projected/2d98a290-3f4a-4467-ab4d-623a42397144-kube-api-access-6vdtz\") pod \"2d98a290-3f4a-4467-ab4d-623a42397144\" (UID: \"2d98a290-3f4a-4467-ab4d-623a42397144\") "
Oct 30 01:15:19.712405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.712353    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58404bec-1cc1-46cc-a5e7-a6994b9c1163-kube-api-access-jmpbr\" (UniqueName: \"kubernetes.io/projected/58404bec-1cc1-46cc-a5e7-a6994b9c1163-kube-api-access-jmpbr\") pod \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\" (UID: \"58404bec-1cc1-46cc-a5e7-a6994b9c1163\") "
Oct 30 01:15:19.712515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.712453    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d98a290-3f4a-4467-ab4d-623a42397144-maas-api-service-ca" pod "2d98a290-3f4a-4467-ab4d-623a42397144" (UID: "2d98a290-3f4a-4467-ab4d-623a42397144"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:19.712573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.712547    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff10f99-a41c-46a1-9dc3-5824e57cf03b-maas-api-service-ca" pod "fff10f99-a41c-46a1-9dc3-5824e57cf03b" (UID: "fff10f99-a41c-46a1-9dc3-5824e57cf03b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:19.712958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.712933    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff10f99-a41c-46a1-9dc3-5824e57cf03b-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:19.712958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.712960    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d98a290-3f4a-4467-ab4d-623a42397144-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:19.714988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.714961    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d98a290-3f4a-4467-ab4d-623a42397144-kube-api-access-6vdtz" pod "2d98a290-3f4a-4467-ab4d-623a42397144" (UID: "2d98a290-3f4a-4467-ab4d-623a42397144"). InnerVolumeSpecName "kube-api-access-6vdtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:19.715169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.715019    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58404bec-1cc1-46cc-a5e7-a6994b9c1163-kube-api-access-jmpbr" pod "58404bec-1cc1-46cc-a5e7-a6994b9c1163" (UID: "58404bec-1cc1-46cc-a5e7-a6994b9c1163"). InnerVolumeSpecName "kube-api-access-jmpbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:19.715169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.715038    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff10f99-a41c-46a1-9dc3-5824e57cf03b-kube-api-access-vvbxs" pod "fff10f99-a41c-46a1-9dc3-5824e57cf03b" (UID: "fff10f99-a41c-46a1-9dc3-5824e57cf03b"). InnerVolumeSpecName "kube-api-access-vvbxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:19.814371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.814327    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvbxs\" (UniqueName: \"kubernetes.io/projected/fff10f99-a41c-46a1-9dc3-5824e57cf03b-kube-api-access-vvbxs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:19.814371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.814366    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdtz\" (UniqueName: \"kubernetes.io/projected/2d98a290-3f4a-4467-ab4d-623a42397144-kube-api-access-6vdtz\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:19.814562 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:19.814381    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpbr\" (UniqueName: \"kubernetes.io/projected/58404bec-1cc1-46cc-a5e7-a6994b9c1163-kube-api-access-jmpbr\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.083147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.083115    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:20.114740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.114704    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" path="/var/lib/kubelet/pods/497a3e39-1661-457e-8fc4-073c263f2f2c/volumes"
Oct 30 01:15:20.322549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.322512    4576 generic.go:356] "Generic (PLEG): container finished" podID="818ea717-759d-4697-9367-6ede754cdc48" containerID="2ef831b8c4cbf50c4d64f2e6ee425fd4dbfa9dc91152b0437481fa353e1c0fa2" exitCode=60
Oct 30 01:15:20.323075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.322548    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" event={"ID":"818ea717-759d-4697-9367-6ede754cdc48","Type":"ContainerDied","Data":"2ef831b8c4cbf50c4d64f2e6ee425fd4dbfa9dc91152b0437481fa353e1c0fa2"}
Oct 30 01:15:20.323075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.322616    4576 scope.go:122] "RemoveContainer" containerID="db69fec296b84ca20f9d3cf35d93fcc3013501906b9c45fba7f7f123090b81b3"
Oct 30 01:15:20.323075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.323034    4576 scope.go:122] "RemoveContainer" containerID="2ef831b8c4cbf50c4d64f2e6ee425fd4dbfa9dc91152b0437481fa353e1c0fa2"
Oct 30 01:15:20.323410 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:20.323387    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx_odh-ai-gateway-infra(818ea717-759d-4697-9367-6ede754cdc48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" podUID="818ea717-759d-4697-9367-6ede754cdc48"
Oct 30 01:15:20.325490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.325471    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"
Oct 30 01:15:20.325576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.325504    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww" event={"ID":"fff10f99-a41c-46a1-9dc3-5824e57cf03b","Type":"ContainerDied","Data":"f7b044127ee35ef1d43a737ca373a431dda058c14a210a2c8b73fd7aeb413237"}
Oct 30 01:15:20.327115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.327096    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"
Oct 30 01:15:20.327259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.327142    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm" event={"ID":"2d98a290-3f4a-4467-ab4d-623a42397144","Type":"ContainerDied","Data":"4b430f21d3e49b163059514178805815b5f9a53823a111c2c9265aacb1578c58"}
Oct 30 01:15:20.336509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.336484    4576 generic.go:356] "Generic (PLEG): container finished" podID="e583bfd6-926a-47a6-9885-74675eac92d0" containerID="dec17c6fc3a3e5cd217921e8a5838e0fb615a8c97c80029edf4e150a01168264" exitCode=60
Oct 30 01:15:20.336643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.336566    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" event={"ID":"e583bfd6-926a-47a6-9885-74675eac92d0","Type":"ContainerDied","Data":"dec17c6fc3a3e5cd217921e8a5838e0fb615a8c97c80029edf4e150a01168264"}
Oct 30 01:15:20.337033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.337012    4576 scope.go:122] "RemoveContainer" containerID="dec17c6fc3a3e5cd217921e8a5838e0fb615a8c97c80029edf4e150a01168264"
Oct 30 01:15:20.337397 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:20.337358    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl_odh-ai-gateway-infra(e583bfd6-926a-47a6-9885-74675eac92d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" podUID="e583bfd6-926a-47a6-9885-74675eac92d0"
Oct 30 01:15:20.339379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.339350    4576 generic.go:356] "Generic (PLEG): container finished" podID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerID="67acc3f21e0d96e09422b6247cdf945039a2f1415e3c673d1a1618b4c5a93e51" exitCode=60
Oct 30 01:15:20.339500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.339441    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" event={"ID":"e7290ded-d2a3-49d1-b712-dc57bac30f01","Type":"ContainerDied","Data":"67acc3f21e0d96e09422b6247cdf945039a2f1415e3c673d1a1618b4c5a93e51"}
Oct 30 01:15:20.339847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.339830    4576 scope.go:122] "RemoveContainer" containerID="67acc3f21e0d96e09422b6247cdf945039a2f1415e3c673d1a1618b4c5a93e51"
Oct 30 01:15:20.339911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.339891    4576 scope.go:122] "RemoveContainer" containerID="5b7209e1a82bc8c592b4e15e0153d877075e3aa3efd52f86bc0eebd1e00a87a5"
Oct 30 01:15:20.340196 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:20.340172    4576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp_odh-ai-gateway-infra(e7290ded-d2a3-49d1-b712-dc57bac30f01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01"
Oct 30 01:15:20.341193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.341159    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v" event={"ID":"58404bec-1cc1-46cc-a5e7-a6994b9c1163","Type":"ContainerDied","Data":"3eb29e8d9ceb926d4a2246e4157bc35496dc4fbcfe21bfbd8ea42f3ccb9e9526"}
Oct 30 01:15:20.341323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.341165    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"
Oct 30 01:15:20.387628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.387501    4576 scope.go:122] "RemoveContainer" containerID="2465222888add22f01cb19deeb55b76bf16a3ef30f3459c08a95f7f586724e3c"
Oct 30 01:15:20.437052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.437021    4576 scope.go:122] "RemoveContainer" containerID="998f3107bbedd05d3aa7a6c1225faac5af5c1d2d8fce88ba55f6c3aa593df3ae"
Oct 30 01:15:20.460903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.460877    4576 scope.go:122] "RemoveContainer" containerID="f4058ce7a63b66fee505e6e0ced855dc3af422b9767648a9dc0b8e77b5e64a8f"
Oct 30 01:15:20.529490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.529463    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"]
Oct 30 01:15:20.539869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.539842    4576 scope.go:122] "RemoveContainer" containerID="c986649431413d911789e270854772718c6b9c11ed2362c9b0a561166e4fd310"
Oct 30 01:15:20.541187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.541163    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43d152-29888715-lrsww"]
Oct 30 01:15:20.545748 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.545318    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:20.613301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.609777    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"]
Oct 30 01:15:20.624091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.624028    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"]
Oct 30 01:15:20.635966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.635269    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-da76a2-29888715-xkwtm"]
Oct 30 01:15:20.648373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.648354    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:20.659180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.659145    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:20.661830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.661812    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:20.741088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.741046    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-kube-api-access-j4hg4\" (UniqueName: \"kubernetes.io/projected/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-kube-api-access-j4hg4\") pod \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\" (UID: \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\") "
Oct 30 01:15:20.741274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.741158    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-maas-api-service-ca\") pod \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\" (UID: \"5e84f94a-3cd7-48ab-844d-e0605f5a16b2\") "
Oct 30 01:15:20.741564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.741542    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-maas-api-service-ca" pod "5e84f94a-3cd7-48ab-844d-e0605f5a16b2" (UID: "5e84f94a-3cd7-48ab-844d-e0605f5a16b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.741938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.741914    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.743753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.743730    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-kube-api-access-j4hg4" pod "5e84f94a-3cd7-48ab-844d-e0605f5a16b2" (UID: "5e84f94a-3cd7-48ab-844d-e0605f5a16b2"). InnerVolumeSpecName "kube-api-access-j4hg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.796387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.796349    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"]
Oct 30 01:15:20.801104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.801078    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b312f7-29888715-76v9v"]
Oct 30 01:15:20.816657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.816629    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"]
Oct 30 01:15:20.861176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.861138    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36763be6-dedf-4cc7-b154-430d488e2b33-kube-api-access-lmwbk\" (UniqueName: \"kubernetes.io/projected/36763be6-dedf-4cc7-b154-430d488e2b33-kube-api-access-lmwbk\") pod \"36763be6-dedf-4cc7-b154-430d488e2b33\" (UID: \"36763be6-dedf-4cc7-b154-430d488e2b33\") "
Oct 30 01:15:20.861383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.861250    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-maas-api-service-ca\") pod \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\" (UID: \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\") "
Oct 30 01:15:20.861383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.861306    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cd525e-0953-419a-9f8b-f9b3c1318918-kube-api-access-46bb9\" (UniqueName: \"kubernetes.io/projected/54cd525e-0953-419a-9f8b-f9b3c1318918-kube-api-access-46bb9\") pod \"54cd525e-0953-419a-9f8b-f9b3c1318918\" (UID: \"54cd525e-0953-419a-9f8b-f9b3c1318918\") "
Oct 30 01:15:20.861383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.861374    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-kube-api-access-j6zm5\" (UniqueName: \"kubernetes.io/projected/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-kube-api-access-j6zm5\") pod \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\" (UID: \"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85\") "
Oct 30 01:15:20.861568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.861443    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36763be6-dedf-4cc7-b154-430d488e2b33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36763be6-dedf-4cc7-b154-430d488e2b33-maas-api-service-ca\") pod \"36763be6-dedf-4cc7-b154-430d488e2b33\" (UID: \"36763be6-dedf-4cc7-b154-430d488e2b33\") "
Oct 30 01:15:20.861637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.861584    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54cd525e-0953-419a-9f8b-f9b3c1318918-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cd525e-0953-419a-9f8b-f9b3c1318918-maas-api-service-ca\") pod \"54cd525e-0953-419a-9f8b-f9b3c1318918\" (UID: \"54cd525e-0953-419a-9f8b-f9b3c1318918\") "
Oct 30 01:15:20.861812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.861725    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-maas-api-service-ca" pod "5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" (UID: "5f3cfe9b-07b2-4a74-a04d-40a8f55efa85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.861918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.861891    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36763be6-dedf-4cc7-b154-430d488e2b33-maas-api-service-ca" pod "36763be6-dedf-4cc7-b154-430d488e2b33" (UID: "36763be6-dedf-4cc7-b154-430d488e2b33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.862133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.862076    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cd525e-0953-419a-9f8b-f9b3c1318918-maas-api-service-ca" pod "54cd525e-0953-419a-9f8b-f9b3c1318918" (UID: "54cd525e-0953-419a-9f8b-f9b3c1318918"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.862133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.862116    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.862305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.862154    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4hg4\" (UniqueName: \"kubernetes.io/projected/5e84f94a-3cd7-48ab-844d-e0605f5a16b2-kube-api-access-j4hg4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.864220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.864169    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36763be6-dedf-4cc7-b154-430d488e2b33-kube-api-access-lmwbk" pod "36763be6-dedf-4cc7-b154-430d488e2b33" (UID: "36763be6-dedf-4cc7-b154-430d488e2b33"). InnerVolumeSpecName "kube-api-access-lmwbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.864220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.864194    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cd525e-0953-419a-9f8b-f9b3c1318918-kube-api-access-46bb9" pod "54cd525e-0953-419a-9f8b-f9b3c1318918" (UID: "54cd525e-0953-419a-9f8b-f9b3c1318918"). InnerVolumeSpecName "kube-api-access-46bb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.865928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.865904    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-kube-api-access-j6zm5" pod "5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" (UID: "5f3cfe9b-07b2-4a74-a04d-40a8f55efa85"). InnerVolumeSpecName "kube-api-access-j6zm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.967276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.964073    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cd525e-0953-419a-9f8b-f9b3c1318918-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.967276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.964110    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmwbk\" (UniqueName: \"kubernetes.io/projected/36763be6-dedf-4cc7-b154-430d488e2b33-kube-api-access-lmwbk\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.967276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.964126    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bb9\" (UniqueName: \"kubernetes.io/projected/54cd525e-0953-419a-9f8b-f9b3c1318918-kube-api-access-46bb9\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.967276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.964140    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6zm5\" (UniqueName: \"kubernetes.io/projected/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85-kube-api-access-j6zm5\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.967276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:20.964152    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36763be6-dedf-4cc7-b154-430d488e2b33-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.357079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.357009    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"
Oct 30 01:15:21.357079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.357030    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf" event={"ID":"36763be6-dedf-4cc7-b154-430d488e2b33","Type":"ContainerDied","Data":"2e12b6255cc1d9f19fb7a6903cf57f9ae733305429ab7b72ab9b15c6d4dc54c4"}
Oct 30 01:15:21.357648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.357094    4576 scope.go:122] "RemoveContainer" containerID="57dab9bbf3ce768b426012064276affa0ca30d33b885bb7aa1e6272a34b38b97"
Oct 30 01:15:21.365176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.365155    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"
Oct 30 01:15:21.365176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.365159    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l" event={"ID":"5f3cfe9b-07b2-4a74-a04d-40a8f55efa85","Type":"ContainerDied","Data":"df97ae66020e95a5ffdc64215fbececdc73f9effc40ba871c3d35e92af9ade8b"}
Oct 30 01:15:21.368015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.367986    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8" event={"ID":"54cd525e-0953-419a-9f8b-f9b3c1318918","Type":"ContainerDied","Data":"ebfb6ba4d593ba2e806cc3d04382970cb9058b4789f1d628a84694c0f20ac258"}
Oct 30 01:15:21.368160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.368093    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"
Oct 30 01:15:21.374279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.371579    4576 scope.go:122] "RemoveContainer" containerID="1b5d5b57bcd0552ad3b0974aaaf942e229eed6eae6d85023622b6960d7e38a5e"
Oct 30 01:15:21.374279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.371738    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"
Oct 30 01:15:21.374279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.371738    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85" event={"ID":"5e84f94a-3cd7-48ab-844d-e0605f5a16b2","Type":"ContainerDied","Data":"4a49c51264b536a7aa64c70494912e61410fae55cb7206f1df2cafcb74aa25dc"}
Oct 30 01:15:21.390812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.390774    4576 scope.go:122] "RemoveContainer" containerID="f307edade830cb5f39696a7527e765cdce8b99e519a7df62da87d747a8490bd8"
Oct 30 01:15:21.406859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.406832    4576 scope.go:122] "RemoveContainer" containerID="e8f60380206047e6e8880159597cb3b8a8a9935b935ab808cf8f479dafb3c5ef"
Oct 30 01:15:21.454083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.454046    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"]
Oct 30 01:15:21.458620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.458583    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3c2596-29888715-2dnrf"]
Oct 30 01:15:21.501274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.500147    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:21.554639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.552621    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"]
Oct 30 01:15:21.558564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.558535    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7d2b75-29888715-l9s5l"]
Oct 30 01:15:21.563825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.563801    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:21.590649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.590626    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"]
Oct 30 01:15:21.598556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.598530    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-d0ae0d-29888715-xrzq8"]
Oct 30 01:15:21.601122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.601106    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:21.615627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.615599    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:21.672854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.672547    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818ea717-759d-4697-9367-6ede754cdc48-kube-api-access-hhq6m\" (UniqueName: \"kubernetes.io/projected/818ea717-759d-4697-9367-6ede754cdc48-kube-api-access-hhq6m\") pod \"818ea717-759d-4697-9367-6ede754cdc48\" (UID: \"818ea717-759d-4697-9367-6ede754cdc48\") "
Oct 30 01:15:21.672854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.672657    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7290ded-d2a3-49d1-b712-dc57bac30f01-kube-api-access-rbbwv\" (UniqueName: \"kubernetes.io/projected/e7290ded-d2a3-49d1-b712-dc57bac30f01-kube-api-access-rbbwv\") pod \"e7290ded-d2a3-49d1-b712-dc57bac30f01\" (UID: \"e7290ded-d2a3-49d1-b712-dc57bac30f01\") "
Oct 30 01:15:21.672854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.672692    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818ea717-759d-4697-9367-6ede754cdc48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818ea717-759d-4697-9367-6ede754cdc48-maas-api-service-ca\") pod \"818ea717-759d-4697-9367-6ede754cdc48\" (UID: \"818ea717-759d-4697-9367-6ede754cdc48\") "
Oct 30 01:15:21.672854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.672805    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7290ded-d2a3-49d1-b712-dc57bac30f01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7290ded-d2a3-49d1-b712-dc57bac30f01-maas-api-service-ca\") pod \"e7290ded-d2a3-49d1-b712-dc57bac30f01\" (UID: \"e7290ded-d2a3-49d1-b712-dc57bac30f01\") "
Oct 30 01:15:21.675863 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.675833    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7290ded-d2a3-49d1-b712-dc57bac30f01-maas-api-service-ca" pod "e7290ded-d2a3-49d1-b712-dc57bac30f01" (UID: "e7290ded-d2a3-49d1-b712-dc57bac30f01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:21.678731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.678696    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818ea717-759d-4697-9367-6ede754cdc48-kube-api-access-hhq6m" pod "818ea717-759d-4697-9367-6ede754cdc48" (UID: "818ea717-759d-4697-9367-6ede754cdc48"). InnerVolumeSpecName "kube-api-access-hhq6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.682989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.682958    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7290ded-d2a3-49d1-b712-dc57bac30f01-kube-api-access-rbbwv" pod "e7290ded-d2a3-49d1-b712-dc57bac30f01" (UID: "e7290ded-d2a3-49d1-b712-dc57bac30f01"). InnerVolumeSpecName "kube-api-access-rbbwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.683378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.683349    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818ea717-759d-4697-9367-6ede754cdc48-maas-api-service-ca" pod "818ea717-759d-4697-9367-6ede754cdc48" (UID: "818ea717-759d-4697-9367-6ede754cdc48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:21.696351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.696321    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"]
Oct 30 01:15:21.700670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.700643    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"]
Oct 30 01:15:21.706031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.705989    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5a344d-29888715-cmw85"]
Oct 30 01:15:21.746291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.744974    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:21.778262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.775564    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7290ded-d2a3-49d1-b712-dc57bac30f01-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.778262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.775596    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhq6m\" (UniqueName: \"kubernetes.io/projected/818ea717-759d-4697-9367-6ede754cdc48-kube-api-access-hhq6m\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.778262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.775769    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbwv\" (UniqueName: \"kubernetes.io/projected/e7290ded-d2a3-49d1-b712-dc57bac30f01-kube-api-access-rbbwv\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.778262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.775786    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818ea717-759d-4697-9367-6ede754cdc48-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.881628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:21.881313    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:22.026751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.026724    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:22.079121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.079089    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e583bfd6-926a-47a6-9885-74675eac92d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e583bfd6-926a-47a6-9885-74675eac92d0-maas-api-service-ca\") pod \"e583bfd6-926a-47a6-9885-74675eac92d0\" (UID: \"e583bfd6-926a-47a6-9885-74675eac92d0\") "
Oct 30 01:15:22.079329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.079156    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e583bfd6-926a-47a6-9885-74675eac92d0-kube-api-access-bpv46\" (UniqueName: \"kubernetes.io/projected/e583bfd6-926a-47a6-9885-74675eac92d0-kube-api-access-bpv46\") pod \"e583bfd6-926a-47a6-9885-74675eac92d0\" (UID: \"e583bfd6-926a-47a6-9885-74675eac92d0\") "
Oct 30 01:15:22.079521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.079496    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e583bfd6-926a-47a6-9885-74675eac92d0-maas-api-service-ca" pod "e583bfd6-926a-47a6-9885-74675eac92d0" (UID: "e583bfd6-926a-47a6-9885-74675eac92d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:22.080029 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.079917    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e583bfd6-926a-47a6-9885-74675eac92d0-maas-api-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.081821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.081789    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e583bfd6-926a-47a6-9885-74675eac92d0-kube-api-access-bpv46" pod "e583bfd6-926a-47a6-9885-74675eac92d0" (UID: "e583bfd6-926a-47a6-9885-74675eac92d0"). InnerVolumeSpecName "kube-api-access-bpv46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:22.110277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.110211    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" path="/var/lib/kubelet/pods/2d98a290-3f4a-4467-ab4d-623a42397144/volumes"
Oct 30 01:15:22.110913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.110888    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" path="/var/lib/kubelet/pods/36763be6-dedf-4cc7-b154-430d488e2b33/volumes"
Oct 30 01:15:22.111574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.111552    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" path="/var/lib/kubelet/pods/54cd525e-0953-419a-9f8b-f9b3c1318918/volumes"
Oct 30 01:15:22.112561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.112537    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" path="/var/lib/kubelet/pods/58404bec-1cc1-46cc-a5e7-a6994b9c1163/volumes"
Oct 30 01:15:22.113112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.113091    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" path="/var/lib/kubelet/pods/5e84f94a-3cd7-48ab-844d-e0605f5a16b2/volumes"
Oct 30 01:15:22.113680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.113658    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" path="/var/lib/kubelet/pods/5f3cfe9b-07b2-4a74-a04d-40a8f55efa85/volumes"
Oct 30 01:15:22.114661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.114639    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" path="/var/lib/kubelet/pods/fff10f99-a41c-46a1-9dc3-5824e57cf03b/volumes"
Oct 30 01:15:22.181675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.181584    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpv46\" (UniqueName: \"kubernetes.io/projected/e583bfd6-926a-47a6-9885-74675eac92d0-kube-api-access-bpv46\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.385056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.384970    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"
Oct 30 01:15:22.385056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.385005    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx" event={"ID":"818ea717-759d-4697-9367-6ede754cdc48","Type":"ContainerDied","Data":"d18654b95ed63e7fb0b6e5578430ae5636bd8a6df80a19051166f985c0f6528d"}
Oct 30 01:15:22.385622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.385097    4576 scope.go:122] "RemoveContainer" containerID="2ef831b8c4cbf50c4d64f2e6ee425fd4dbfa9dc91152b0437481fa353e1c0fa2"
Oct 30 01:15:22.388165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.388145    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"
Oct 30 01:15:22.388342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.388143    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl" event={"ID":"e583bfd6-926a-47a6-9885-74675eac92d0","Type":"ContainerDied","Data":"fe967cd171539f18a1229fdfb05f1915f4038dd6977c77e869f3aa8152049eb9"}
Oct 30 01:15:22.389832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.389814    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"
Oct 30 01:15:22.390170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.389841    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp" event={"ID":"e7290ded-d2a3-49d1-b712-dc57bac30f01","Type":"ContainerDied","Data":"d2775b6b21de2b081d4db709e626802b4e30ecfa41c860afb996e5fee2b03126"}
Oct 30 01:15:22.397777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.397757    4576 scope.go:122] "RemoveContainer" containerID="dec17c6fc3a3e5cd217921e8a5838e0fb615a8c97c80029edf4e150a01168264"
Oct 30 01:15:22.407527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.407509    4576 scope.go:122] "RemoveContainer" containerID="67acc3f21e0d96e09422b6247cdf945039a2f1415e3c673d1a1618b4c5a93e51"
Oct 30 01:15:22.442787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.442759    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"]
Oct 30 01:15:22.446034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.446012    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-42d56b-29888715-zbccx"]
Oct 30 01:15:22.482417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.482388    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"]
Oct 30 01:15:22.487667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.487642    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-bdf6f3-29888715-r4wxp"]
Oct 30 01:15:22.520192 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.520154    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"]
Oct 30 01:15:22.523322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:22.523299    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-298ddc-29888715-chxbl"]
Oct 30 01:15:23.253631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:23.253597    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:24.110434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:24.110402    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818ea717-759d-4697-9367-6ede754cdc48" path="/var/lib/kubelet/pods/818ea717-759d-4697-9367-6ede754cdc48/volumes"
Oct 30 01:15:24.110864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:24.110840    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" path="/var/lib/kubelet/pods/e583bfd6-926a-47a6-9885-74675eac92d0/volumes"
Oct 30 01:15:24.111253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:24.111224    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" path="/var/lib/kubelet/pods/e7290ded-d2a3-49d1-b712-dc57bac30f01/volumes"
Oct 30 01:15:24.871324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:24.871284    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:25.537826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:25.537785    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:25.550836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:25.550804    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:25.997222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:25.997184    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:15:26.439813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:26.439776    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:15:26.484692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:26.484655    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:28.481760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:28.481720    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:30.149511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:30.149471    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:30.379022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:30.378984    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:31.908373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:31.908338    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:33.660004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:33.659955    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:35.229608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:35.229562    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:36.851964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:36.851921    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:37.397349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:37.397311    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:38.614411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:38.614375    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:38.871608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:38.871512    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:38.999264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:38.999200    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:39.891818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:39.891785    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:40.412874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:40.412835    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:41.028411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:41.028372    4576 scope.go:122] "RemoveContainer" containerID="06bc3fd66e8d8034b6cce4926bbd99ab4ba65837bd1bf174c0589ff3cf2e5b62"
Oct 30 01:15:41.499775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:41.499742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:42.311817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:42.311785    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:44.068141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:44.068103    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:45.771041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:45.770998    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:47.334864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:47.334825    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:48.384947 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:15:48.384914    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:15:48.798026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:48.797991    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:49.179739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.179659    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"]
Oct 30 01:15:49.180379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180359    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:49.180379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180379    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180397    4576 cpu_manager.go:414] "containerMap: removing container" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180404    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180415    4576 cpu_manager.go:414] "containerMap: removing container" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180422    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180433    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180439    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180448    4576 cpu_manager.go:414] "containerMap: removing container" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180454    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180466    4576 cpu_manager.go:414] "containerMap: removing container" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180474    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180483    4576 cpu_manager.go:414] "containerMap: removing container" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180490    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180512    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180518    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180529    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180536    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180550    4576 cpu_manager.go:414] "containerMap: removing container" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180557    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:49.180563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180569    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180577    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180588    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180595    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180605    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180612    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180626    4576 cpu_manager.go:414] "containerMap: removing container" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180633    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180647    4576 cpu_manager.go:414] "containerMap: removing container" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180655    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180665    4576 cpu_manager.go:414] "containerMap: removing container" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180671    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180682    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180689    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180697    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180704    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180715    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180721    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180730    4576 cpu_manager.go:414] "containerMap: removing container" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180736    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180754    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180762    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180777    4576 cpu_manager.go:414] "containerMap: removing container" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180784    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180800    4576 cpu_manager.go:414] "containerMap: removing container" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180808    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180821    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180828    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180838    4576 cpu_manager.go:414] "containerMap: removing container" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180845    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180854    4576 cpu_manager.go:414] "containerMap: removing container" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180860    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180874    4576 cpu_manager.go:414] "containerMap: removing container" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.180881    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181029    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181045    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181056    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181063    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181071    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181081    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181090    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181100    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181107    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181115    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181123    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d98a290-3f4a-4467-ab4d-623a42397144" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181133    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181143    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181153    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181161    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181168    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181178    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181188    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181195    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181204    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181214    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff10f99-a41c-46a1-9dc3-5824e57cf03b" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181223    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181254    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181297    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181308    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181316    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:49.181576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.181328    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:49.187730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.187705    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.203546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.203522    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"]
Oct 30 01:15:49.211061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.211037    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-24d5ce\"/\"default-dockercfg-sscjg\""
Oct 30 01:15:49.211201 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.211174    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-24d5ce\"/\"auth-test-model-24d5ce-kserve-self-signed-certs\""
Oct 30 01:15:49.217864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.217838    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-24d5ce\"/\"kube-root-ca.crt\""
Oct 30 01:15:49.217983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.217868    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-24d5ce\"/\"openshift-service-ca.crt\""
Oct 30 01:15:49.259284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.259210    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:49.289178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.289134    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4jg\" (UniqueName: \"kubernetes.io/projected/2e4fc1ab-ec7a-4056-a97d-220d8527b083-kube-api-access-7j4jg\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.289379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.289187    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-model-cache\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.289379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.289318    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tls-certs\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.289533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.289448    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-dshm\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.289533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.289514    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tmp-dir\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.289648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.289621    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-home\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.390671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.390637    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-home\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.391170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.390695    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4jg\" (UniqueName: \"kubernetes.io/projected/2e4fc1ab-ec7a-4056-a97d-220d8527b083-kube-api-access-7j4jg\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.391170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.390756    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-model-cache\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.391170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.390812    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tls-certs\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.391170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.390921    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-dshm\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.391170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.391006    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tmp-dir\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.391170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.391101    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-home\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.391524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.391427    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tmp-dir\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.391524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.391454    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-model-cache\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.394163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.394133    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-dshm\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.394692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.394669    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tls-certs\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.437064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.436955    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4jg\" (UniqueName: \"kubernetes.io/projected/2e4fc1ab-ec7a-4056-a97d-220d8527b083-kube-api-access-7j4jg\") pod \"auth-test-model-24d5ce-kserve-7784766b97-gj2c2\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.512102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.512067    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:49.840442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:49.840415    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"]
Oct 30 01:15:49.841621 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:49.841596    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e4fc1ab_ec7a_4056_a97d_220d8527b083.slice/crio-b3f57627dec6164187b478d0502fb92d7a2a35453499c86ff2233213e2800655 WatchSource:0}: Error finding container b3f57627dec6164187b478d0502fb92d7a2a35453499c86ff2233213e2800655: Status 404 returned error can't find the container with id b3f57627dec6164187b478d0502fb92d7a2a35453499c86ff2233213e2800655
Oct 30 01:15:50.562500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:50.562458    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:50.691543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:50.691505    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2" event={"ID":"2e4fc1ab-ec7a-4056-a97d-220d8527b083","Type":"ContainerStarted","Data":"8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641"}
Oct 30 01:15:50.691543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:50.691547    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2" event={"ID":"2e4fc1ab-ec7a-4056-a97d-220d8527b083","Type":"ContainerStarted","Data":"b3f57627dec6164187b478d0502fb92d7a2a35453499c86ff2233213e2800655"}
Oct 30 01:15:50.736341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:50.736291    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2" podStartSLOduration=1.736278404 podStartE2EDuration="1.736278404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:50.732711615 +0000 UTC m=+2230.804035452" watchObservedRunningTime="2026-10-30 01:15:50.736278404 +0000 UTC m=+2230.807602239"
Oct 30 01:15:51.513169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.513132    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:51.986926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.986884    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"]
Oct 30 01:15:51.987688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987668    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:51.987688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987689    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987721    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987726    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987738    4576 cpu_manager.go:414] "containerMap: removing container" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987743    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987751    4576 cpu_manager.go:414] "containerMap: removing container" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987756    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497a3e39-1661-457e-8fc4-073c263f2f2c" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987762    4576 cpu_manager.go:414] "containerMap: removing container" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987767    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36763be6-dedf-4cc7-b154-430d488e2b33" containerName="cleanup"
Oct 30 01:15:51.987782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987784    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:51.988083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987789    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e583bfd6-926a-47a6-9885-74675eac92d0" containerName="cleanup"
Oct 30 01:15:51.988083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987870    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="58404bec-1cc1-46cc-a5e7-a6994b9c1163" containerName="cleanup"
Oct 30 01:15:51.988083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987880    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="818ea717-759d-4697-9367-6ede754cdc48" containerName="cleanup"
Oct 30 01:15:51.988083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987886    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e84f94a-3cd7-48ab-844d-e0605f5a16b2" containerName="cleanup"
Oct 30 01:15:51.988083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987894    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3cfe9b-07b2-4a74-a04d-40a8f55efa85" containerName="cleanup"
Oct 30 01:15:51.988083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987903    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cd525e-0953-419a-9f8b-f9b3c1318918" containerName="cleanup"
Oct 30 01:15:51.988083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.987910    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7290ded-d2a3-49d1-b712-dc57bac30f01" containerName="cleanup"
Oct 30 01:15:51.991649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:51.991633    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.007881 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.007858    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:15:52.008859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.008837    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"]
Oct 30 01:15:52.018081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.018052    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:15:52.022287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.022255    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.022481 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.022333    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-726vq\" (UniqueName: \"kubernetes.io/projected/96d6205c-1d2f-4325-a056-66442ca51f58-kube-api-access-726vq\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.022625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.022487    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.059862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.059827    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:52.124068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.124030    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.124453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.124181    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.124453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.124315    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-726vq\" (UniqueName: \"kubernetes.io/projected/96d6205c-1d2f-4325-a056-66442ca51f58-kube-api-access-726vq\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.126134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.125777    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.139894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.139861    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.146034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.146002    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-726vq\" (UniqueName: \"kubernetes.io/projected/96d6205c-1d2f-4325-a056-66442ca51f58-kube-api-access-726vq\") pod \"maas-api-revoke-sub-ac5602db-bjs42\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.307856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.307773    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:52.610016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.609983    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"]
Oct 30 01:15:52.610642 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:15:52.610607    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d6205c_1d2f_4325_a056_66442ca51f58.slice/crio-2a5867dd7eb06114f92a57118e263809e18fb7c3146c51873306a5d92f50e258 WatchSource:0}: Error finding container 2a5867dd7eb06114f92a57118e263809e18fb7c3146c51873306a5d92f50e258: Status 404 returned error can't find the container with id 2a5867dd7eb06114f92a57118e263809e18fb7c3146c51873306a5d92f50e258
Oct 30 01:15:52.707317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:52.707282    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42" event={"ID":"96d6205c-1d2f-4325-a056-66442ca51f58","Type":"ContainerStarted","Data":"2a5867dd7eb06114f92a57118e263809e18fb7c3146c51873306a5d92f50e258"}
Oct 30 01:15:53.530747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:53.530709    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:53.716692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:53.716654    4576 generic.go:356] "Generic (PLEG): container finished" podID="96d6205c-1d2f-4325-a056-66442ca51f58" containerID="9fadee815c72a22d9c947fe3cd8e9154105291fa3765783798cf813278b9df2d" exitCode=0
Oct 30 01:15:53.716887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:53.716688    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42" event={"ID":"96d6205c-1d2f-4325-a056-66442ca51f58","Type":"ContainerDied","Data":"9fadee815c72a22d9c947fe3cd8e9154105291fa3765783798cf813278b9df2d"}
Oct 30 01:15:54.903494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:54.903446    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:54.973776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:54.973736    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d6205c-1d2f-4325-a056-66442ca51f58-kube-api-access-726vq\" (UniqueName: \"kubernetes.io/projected/96d6205c-1d2f-4325-a056-66442ca51f58-kube-api-access-726vq\") pod \"96d6205c-1d2f-4325-a056-66442ca51f58\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") "
Oct 30 01:15:54.973991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:54.973924    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-openshift-ca\") pod \"96d6205c-1d2f-4325-a056-66442ca51f58\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") "
Oct 30 01:15:54.974101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:54.974076    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-xks-ca\") pod \"96d6205c-1d2f-4325-a056-66442ca51f58\" (UID: \"96d6205c-1d2f-4325-a056-66442ca51f58\") "
Oct 30 01:15:54.974556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:54.974528    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-openshift-ca" pod "96d6205c-1d2f-4325-a056-66442ca51f58" (UID: "96d6205c-1d2f-4325-a056-66442ca51f58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:54.974805 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:54.974777    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:54.976837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:54.976810    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d6205c-1d2f-4325-a056-66442ca51f58-kube-api-access-726vq" pod "96d6205c-1d2f-4325-a056-66442ca51f58" (UID: "96d6205c-1d2f-4325-a056-66442ca51f58"). InnerVolumeSpecName "kube-api-access-726vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:54.977431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:54.977408    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-xks-ca" pod "96d6205c-1d2f-4325-a056-66442ca51f58" (UID: "96d6205c-1d2f-4325-a056-66442ca51f58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:55.083283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:55.076588    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d6205c-1d2f-4325-a056-66442ca51f58-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:55.083283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:55.076623    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-726vq\" (UniqueName: \"kubernetes.io/projected/96d6205c-1d2f-4325-a056-66442ca51f58-kube-api-access-726vq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:15:55.777255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:55.771709    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42" event={"ID":"96d6205c-1d2f-4325-a056-66442ca51f58","Type":"ContainerDied","Data":"2a5867dd7eb06114f92a57118e263809e18fb7c3146c51873306a5d92f50e258"}
Oct 30 01:15:55.777255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:55.771779    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a5867dd7eb06114f92a57118e263809e18fb7c3146c51873306a5d92f50e258"
Oct 30 01:15:55.777255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:55.771851    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"
Oct 30 01:15:55.946625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:55.946483    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:56.542069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:56.542010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:56.946550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:56.946515    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:57.752251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:57.752207    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:59.407090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.407057    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:15:59.442470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.442083    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"]
Oct 30 01:15:59.442939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.442905    4576 cpu_manager.go:414] "containerMap: removing container" podUID="96d6205c-1d2f-4325-a056-66442ca51f58" containerName="revoke-keys"
Oct 30 01:15:59.442939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.442926    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d6205c-1d2f-4325-a056-66442ca51f58" containerName="revoke-keys"
Oct 30 01:15:59.443100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.443011    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d6205c-1d2f-4325-a056-66442ca51f58" containerName="revoke-keys"
Oct 30 01:15:59.447101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.447080    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.464890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.464842    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"]
Oct 30 01:15:59.484257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.475534    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:15:59.484257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.475680    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:15:59.512867 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.512837    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:59.532521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.532491    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:15:59.556957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.556924    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.557197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.557174    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.557379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.557348    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwzl\" (UniqueName: \"kubernetes.io/projected/02389651-5192-43ce-b05b-43f5a6a484ca-kube-api-access-7jwzl\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.658329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.658245    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.658329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.658309    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwzl\" (UniqueName: \"kubernetes.io/projected/02389651-5192-43ce-b05b-43f5a6a484ca-kube-api-access-7jwzl\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.658560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.658428    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.659125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.659056    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.660765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.660743    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.693823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.693787    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwzl\" (UniqueName: \"kubernetes.io/projected/02389651-5192-43ce-b05b-43f5a6a484ca-kube-api-access-7jwzl\") pod \"maas-api-revoke-sub-863b7fe1-g7k6j\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.818569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.818544    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:15:59.835802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:15:59.835776    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:16:00.151176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:00.151146    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"]
Oct 30 01:16:00.151753 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:00.151728    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02389651_5192_43ce_b05b_43f5a6a484ca.slice/crio-14e171040c81a2730581dfb31fe9096e3ba773fe6b25191bba45a4e7b7590a91 WatchSource:0}: Error finding container 14e171040c81a2730581dfb31fe9096e3ba773fe6b25191bba45a4e7b7590a91: Status 404 returned error can't find the container with id 14e171040c81a2730581dfb31fe9096e3ba773fe6b25191bba45a4e7b7590a91
Oct 30 01:16:00.833906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:00.833868    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j" event={"ID":"02389651-5192-43ce-b05b-43f5a6a484ca","Type":"ContainerStarted","Data":"96e55be0aa0b13f237a5c017b0adaa795eded2af8d3dfd06f13c5858d5f88a30"}
Oct 30 01:16:00.834447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:00.833940    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j" event={"ID":"02389651-5192-43ce-b05b-43f5a6a484ca","Type":"ContainerStarted","Data":"14e171040c81a2730581dfb31fe9096e3ba773fe6b25191bba45a4e7b7590a91"}
Oct 30 01:16:00.896678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:00.894224    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:00.897415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:00.897357    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j" podStartSLOduration=1.8973390860000001 podStartE2EDuration="1.897339086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:00.888501791 +0000 UTC m=+2240.959825626" watchObservedRunningTime="2026-10-30 01:16:00.897339086 +0000 UTC m=+2240.968662923"
Oct 30 01:16:01.844962 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:01.844931    4576 generic.go:356] "Generic (PLEG): container finished" podID="02389651-5192-43ce-b05b-43f5a6a484ca" containerID="96e55be0aa0b13f237a5c017b0adaa795eded2af8d3dfd06f13c5858d5f88a30" exitCode=0
Oct 30 01:16:01.845404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:01.845012    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j" event={"ID":"02389651-5192-43ce-b05b-43f5a6a484ca","Type":"ContainerDied","Data":"96e55be0aa0b13f237a5c017b0adaa795eded2af8d3dfd06f13c5858d5f88a30"}
Oct 30 01:16:02.412749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:02.412696    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:03.028640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.028599    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:16:03.137148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.137072    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02389651-5192-43ce-b05b-43f5a6a484ca-kube-api-access-7jwzl\" (UniqueName: \"kubernetes.io/projected/02389651-5192-43ce-b05b-43f5a6a484ca-kube-api-access-7jwzl\") pod \"02389651-5192-43ce-b05b-43f5a6a484ca\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") "
Oct 30 01:16:03.137312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.137284    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-xks-ca\") pod \"02389651-5192-43ce-b05b-43f5a6a484ca\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") "
Oct 30 01:16:03.137364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.137338    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-openshift-ca\") pod \"02389651-5192-43ce-b05b-43f5a6a484ca\" (UID: \"02389651-5192-43ce-b05b-43f5a6a484ca\") "
Oct 30 01:16:03.138206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.137787    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-openshift-ca" pod "02389651-5192-43ce-b05b-43f5a6a484ca" (UID: "02389651-5192-43ce-b05b-43f5a6a484ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:03.138206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.138156    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:03.139722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.139694    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02389651-5192-43ce-b05b-43f5a6a484ca-kube-api-access-7jwzl" pod "02389651-5192-43ce-b05b-43f5a6a484ca" (UID: "02389651-5192-43ce-b05b-43f5a6a484ca"). InnerVolumeSpecName "kube-api-access-7jwzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:03.140127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.140107    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-xks-ca" pod "02389651-5192-43ce-b05b-43f5a6a484ca" (UID: "02389651-5192-43ce-b05b-43f5a6a484ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:03.240524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.240491    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jwzl\" (UniqueName: \"kubernetes.io/projected/02389651-5192-43ce-b05b-43f5a6a484ca-kube-api-access-7jwzl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:03.240524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.240522    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02389651-5192-43ce-b05b-43f5a6a484ca-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:03.861824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.861792    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j" event={"ID":"02389651-5192-43ce-b05b-43f5a6a484ca","Type":"ContainerDied","Data":"14e171040c81a2730581dfb31fe9096e3ba773fe6b25191bba45a4e7b7590a91"}
Oct 30 01:16:03.861824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.861809    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"
Oct 30 01:16:03.864132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:03.861845    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14e171040c81a2730581dfb31fe9096e3ba773fe6b25191bba45a4e7b7590a91"
Oct 30 01:16:04.060449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:04.060400    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:04.870528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:04.870493    4576 generic.go:356] "Generic (PLEG): container finished" podID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerID="191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a" exitCode=0
Oct 30 01:16:04.870790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:04.870549    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" event={"ID":"9cc90602-3cba-4f39-95ff-d46de1dd32f0","Type":"ContainerDied","Data":"191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a"}
Oct 30 01:16:04.871057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:04.871038    4576 scope.go:122] "RemoveContainer" containerID="191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a"
Oct 30 01:16:04.872762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:04.872732    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:04.872858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:04.872798    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:04.872858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:04.872830    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:05.505218 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:05.505176    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:05.899943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:05.899854    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" event={"ID":"9cc90602-3cba-4f39-95ff-d46de1dd32f0","Type":"ContainerStarted","Data":"c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b"}
Oct 30 01:16:06.592485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:06.592440    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:16:06.593055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:06.593016    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:16:06.593138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:06.593062    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 01:16:07.052291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:07.052216    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:07.592260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:07.591661    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:16:07.592260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:07.591723    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 01:16:07.806286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:07.801905    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:08.552634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:08.552601    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:08.597257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:08.594648    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:16:08.597257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:08.594713    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 01:16:09.521259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.520692    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:09.594259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.591624    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:16:09.594259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.591681    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 01:16:09.655256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.655118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:09.845310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.844914    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"]
Oct 30 01:16:09.845551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.845535    4576 cpu_manager.go:414] "containerMap: removing container" podUID="02389651-5192-43ce-b05b-43f5a6a484ca" containerName="revoke-keys"
Oct 30 01:16:09.845551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.845551    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02389651-5192-43ce-b05b-43f5a6a484ca" containerName="revoke-keys"
Oct 30 01:16:09.845696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.845639    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="02389651-5192-43ce-b05b-43f5a6a484ca" containerName="revoke-keys"
Oct 30 01:16:09.849281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.849257    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:09.867572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.867541    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"]
Oct 30 01:16:09.868458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.868373    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:09.868690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.868657    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:09.868884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.868869    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jwk7\" (UniqueName: \"kubernetes.io/projected/5eecacef-9738-4ee8-b88a-55d582baf4a0-kube-api-access-6jwk7\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:09.907978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.907944    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:16:09.908452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.908428    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:09.909435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.909271    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:09.972147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.972110    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:09.973273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.973249    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:09.973531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.973510    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jwk7\" (UniqueName: \"kubernetes.io/projected/5eecacef-9738-4ee8-b88a-55d582baf4a0-kube-api-access-6jwk7\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:09.974131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:09.973143    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:10.001130 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:10.001086    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:10.001779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:10.001757    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jwk7\" (UniqueName: \"kubernetes.io/projected/5eecacef-9738-4ee8-b88a-55d582baf4a0-kube-api-access-6jwk7\") pod \"maas-api-revoke-sub-cd6ac37b-bs5vk\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:10.164400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:10.164301    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:10.209837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:10.209800    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:10.548784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:10.548750    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"]
Oct 30 01:16:10.598163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:10.597715    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 01:16:10.598163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:10.597771    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 01:16:10.959089 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:10.959049    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk" event={"ID":"5eecacef-9738-4ee8-b88a-55d582baf4a0","Type":"ContainerStarted","Data":"51af3dca5cce8320056335b12c24065bb9ee0b949b3c6c5d567852b6359d97fa"}
Oct 30 01:16:11.597638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:11.597604    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:16:11.598009 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:11.597988    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:16:11.598632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:11.598610    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:16:11.775122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:11.775086    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:11.970081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:11.970045    4576 generic.go:356] "Generic (PLEG): container finished" podID="5eecacef-9738-4ee8-b88a-55d582baf4a0" containerID="f07add62a8170b2368717fe06d59873ce056235023b101512a293540a80aeb9b" exitCode=0
Oct 30 01:16:11.970335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:11.970132    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk" event={"ID":"5eecacef-9738-4ee8-b88a-55d582baf4a0","Type":"ContainerDied","Data":"f07add62a8170b2368717fe06d59873ce056235023b101512a293540a80aeb9b"}
Oct 30 01:16:13.126913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.126885    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:13.143514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.143489    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:13.248555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.248459    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eecacef-9738-4ee8-b88a-55d582baf4a0-kube-api-access-6jwk7\" (UniqueName: \"kubernetes.io/projected/5eecacef-9738-4ee8-b88a-55d582baf4a0-kube-api-access-6jwk7\") pod \"5eecacef-9738-4ee8-b88a-55d582baf4a0\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") "
Oct 30 01:16:13.254070 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.253603    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eecacef-9738-4ee8-b88a-55d582baf4a0-kube-api-access-6jwk7" pod "5eecacef-9738-4ee8-b88a-55d582baf4a0" (UID: "5eecacef-9738-4ee8-b88a-55d582baf4a0"). InnerVolumeSpecName "kube-api-access-6jwk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:13.265935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.254438    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-openshift-ca\") pod \"5eecacef-9738-4ee8-b88a-55d582baf4a0\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") "
Oct 30 01:16:13.265935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.264401    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-xks-ca\") pod \"5eecacef-9738-4ee8-b88a-55d582baf4a0\" (UID: \"5eecacef-9738-4ee8-b88a-55d582baf4a0\") "
Oct 30 01:16:13.265935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.254879    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-openshift-ca" pod "5eecacef-9738-4ee8-b88a-55d582baf4a0" (UID: "5eecacef-9738-4ee8-b88a-55d582baf4a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:13.267354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.267327    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:13.267485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.267357    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jwk7\" (UniqueName: \"kubernetes.io/projected/5eecacef-9738-4ee8-b88a-55d582baf4a0-kube-api-access-6jwk7\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:13.281757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.281721    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-xks-ca" pod "5eecacef-9738-4ee8-b88a-55d582baf4a0" (UID: "5eecacef-9738-4ee8-b88a-55d582baf4a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:13.342988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.342952    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:13.375934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.375338    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eecacef-9738-4ee8-b88a-55d582baf4a0-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:13.480863 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.480826    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:13.798184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.798145    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:13.822887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.822856    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"]
Oct 30 01:16:13.823704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.823682    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5eecacef-9738-4ee8-b88a-55d582baf4a0" containerName="revoke-keys"
Oct 30 01:16:13.823704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.823702    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eecacef-9738-4ee8-b88a-55d582baf4a0" containerName="revoke-keys"
Oct 30 01:16:13.823896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.823788    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eecacef-9738-4ee8-b88a-55d582baf4a0" containerName="revoke-keys"
Oct 30 01:16:13.827910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.827891    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:13.843273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.843216    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"]
Oct 30 01:16:13.975082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.975052    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"]
Oct 30 01:16:13.979533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.979510    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:13.983881 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.983856    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:13.984097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.984079    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:13.984169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.984121    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfkw\" (UniqueName: \"kubernetes.io/projected/01d77e55-3c6b-4ec9-a01b-8c5df878245d-kube-api-access-tqfkw\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:13.988677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.988655    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"
Oct 30 01:16:13.988796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.988679    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk" event={"ID":"5eecacef-9738-4ee8-b88a-55d582baf4a0","Type":"ContainerDied","Data":"51af3dca5cce8320056335b12c24065bb9ee0b949b3c6c5d567852b6359d97fa"}
Oct 30 01:16:13.988796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.988720    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51af3dca5cce8320056335b12c24065bb9ee0b949b3c6c5d567852b6359d97fa"
Oct 30 01:16:13.998174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.998150    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4b973b-openshift-default-dockercfg-6znzx\""
Oct 30 01:16:13.999616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:13.999586    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"]
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.085900    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-certs\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086034    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086109    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-data\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086152    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-envoy\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086194    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086224    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-podinfo\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086377    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-credential-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086460    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f35c47-9409-4997-8a81-3d2ab56fab7f-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086527    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-token\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086558    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc42s\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-kube-api-access-kc42s\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086685    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.086724    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfkw\" (UniqueName: \"kubernetes.io/projected/01d77e55-3c6b-4ec9-a01b-8c5df878245d-kube-api-access-tqfkw\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.087544    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:14.092309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.089301    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:14.110025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.109988    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfkw\" (UniqueName: \"kubernetes.io/projected/01d77e55-3c6b-4ec9-a01b-8c5df878245d-kube-api-access-tqfkw\") pod \"maas-api-revoke-sub-08beaa65-bs5rx\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:14.153556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.153524    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:14.188533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188500    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-data\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.188738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188538    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-envoy\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.188738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188572    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.188738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188598    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-podinfo\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.188738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188644    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-credential-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.188738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188695    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f35c47-9409-4997-8a81-3d2ab56fab7f-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.188738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188735    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-token\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.189194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188762    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc42s\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-kube-api-access-kc42s\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.189194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.188914    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-certs\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.189194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.189037    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-data\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.189194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.189043    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.189466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.189296    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-certs\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.189838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.189784    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f35c47-9409-4997-8a81-3d2ab56fab7f-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.189838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.189836    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-credential-socket\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.191292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.191270    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-envoy\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.191741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.191718    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-podinfo\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.211017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.210984    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-token\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.211017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.211010    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc42s\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-kube-api-access-kc42s\") pod \"e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") " pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.295585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.295548    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:14.461383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.461356    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"]
Oct 30 01:16:14.461916 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:14.461886    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d77e55_3c6b_4ec9_a01b_8c5df878245d.slice/crio-091143e1cabeafa0ac6ac7ac7d222d3a3705606303eabf7bd2f58a6b0b8d02bd WatchSource:0}: Error finding container 091143e1cabeafa0ac6ac7ac7d222d3a3705606303eabf7bd2f58a6b0b8d02bd: Status 404 returned error can't find the container with id 091143e1cabeafa0ac6ac7ac7d222d3a3705606303eabf7bd2f58a6b0b8d02bd
Oct 30 01:16:14.601662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.601598    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"]
Oct 30 01:16:14.602495 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:14.602468    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f35c47_9409_4997_8a81_3d2ab56fab7f.slice/crio-8782c9a60b353bdc4fb8b9f282d83985b1933b6ee1bd5a110a98afbce6d9302b WatchSource:0}: Error finding container 8782c9a60b353bdc4fb8b9f282d83985b1933b6ee1bd5a110a98afbce6d9302b: Status 404 returned error can't find the container with id 8782c9a60b353bdc4fb8b9f282d83985b1933b6ee1bd5a110a98afbce6d9302b
Oct 30 01:16:14.605303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.605271    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:14.605407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.605349    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:14.605407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.605389    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:14.998768 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.998731    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx" event={"ID":"01d77e55-3c6b-4ec9-a01b-8c5df878245d","Type":"ContainerStarted","Data":"f0ea12f71beaaf037a3f828e6a9fa94a9e05035eb3cdae8afe115e54b7cb9072"}
Oct 30 01:16:14.998982 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:14.998798    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx" event={"ID":"01d77e55-3c6b-4ec9-a01b-8c5df878245d","Type":"ContainerStarted","Data":"091143e1cabeafa0ac6ac7ac7d222d3a3705606303eabf7bd2f58a6b0b8d02bd"}
Oct 30 01:16:15.000859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.000807    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" event={"ID":"74f35c47-9409-4997-8a81-3d2ab56fab7f","Type":"ContainerStarted","Data":"6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e"}
Oct 30 01:16:15.001082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.000868    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" event={"ID":"74f35c47-9409-4997-8a81-3d2ab56fab7f","Type":"ContainerStarted","Data":"8782c9a60b353bdc4fb8b9f282d83985b1933b6ee1bd5a110a98afbce6d9302b"}
Oct 30 01:16:15.041485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.041433    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx" podStartSLOduration=2.041410566 podStartE2EDuration="2.041410566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:15.039313803 +0000 UTC m=+2255.110637639" watchObservedRunningTime="2026-10-30 01:16:15.041410566 +0000 UTC m=+2255.112734400"
Oct 30 01:16:15.084729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.084678    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" podStartSLOduration=2.084663184 podStartE2EDuration="2.084663184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:15.083579443 +0000 UTC m=+2255.154903304" watchObservedRunningTime="2026-10-30 01:16:15.084663184 +0000 UTC m=+2255.155987019"
Oct 30 01:16:15.240326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.240225    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:15.298525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.298482    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:15.300050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.300022    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 01:16:15.300254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.300072    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 01:16:15.366253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.366193    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:15.527330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:15.525532    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:16.011322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:16.011282    4576 generic.go:356] "Generic (PLEG): container finished" podID="01d77e55-3c6b-4ec9-a01b-8c5df878245d" containerID="f0ea12f71beaaf037a3f828e6a9fa94a9e05035eb3cdae8afe115e54b7cb9072" exitCode=0
Oct 30 01:16:16.011533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:16.011360    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx" event={"ID":"01d77e55-3c6b-4ec9-a01b-8c5df878245d","Type":"ContainerDied","Data":"f0ea12f71beaaf037a3f828e6a9fa94a9e05035eb3cdae8afe115e54b7cb9072"}
Oct 30 01:16:16.296759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:16.296652    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 01:16:16.296759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:16.296737    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 01:16:17.096263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.088377    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:17.297310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.296974    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 01:16:17.297310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.297030    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 01:16:17.301889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.301865    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:17.361278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.357808    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d77e55-3c6b-4ec9-a01b-8c5df878245d-kube-api-access-tqfkw\" (UniqueName: \"kubernetes.io/projected/01d77e55-3c6b-4ec9-a01b-8c5df878245d-kube-api-access-tqfkw\") pod \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") "
Oct 30 01:16:17.361278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.357964    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-openshift-ca\") pod \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") "
Oct 30 01:16:17.361278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.358092    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-xks-ca\") pod \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\" (UID: \"01d77e55-3c6b-4ec9-a01b-8c5df878245d\") "
Oct 30 01:16:17.361278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.360820    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-openshift-ca" pod "01d77e55-3c6b-4ec9-a01b-8c5df878245d" (UID: "01d77e55-3c6b-4ec9-a01b-8c5df878245d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:17.361278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.361156    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d77e55-3c6b-4ec9-a01b-8c5df878245d-kube-api-access-tqfkw" pod "01d77e55-3c6b-4ec9-a01b-8c5df878245d" (UID: "01d77e55-3c6b-4ec9-a01b-8c5df878245d"). InnerVolumeSpecName "kube-api-access-tqfkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:17.361278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.361227    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:17.374256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.362427    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-xks-ca" pod "01d77e55-3c6b-4ec9-a01b-8c5df878245d" (UID: "01d77e55-3c6b-4ec9-a01b-8c5df878245d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:17.462292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.462223    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqfkw\" (UniqueName: \"kubernetes.io/projected/01d77e55-3c6b-4ec9-a01b-8c5df878245d-kube-api-access-tqfkw\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:17.462292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:17.462292    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d77e55-3c6b-4ec9-a01b-8c5df878245d-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:18.028835 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:18.028804    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"
Oct 30 01:16:18.028835 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:18.028804    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx" event={"ID":"01d77e55-3c6b-4ec9-a01b-8c5df878245d","Type":"ContainerDied","Data":"091143e1cabeafa0ac6ac7ac7d222d3a3705606303eabf7bd2f58a6b0b8d02bd"}
Oct 30 01:16:18.029110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:18.028874    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="091143e1cabeafa0ac6ac7ac7d222d3a3705606303eabf7bd2f58a6b0b8d02bd"
Oct 30 01:16:18.296106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:18.296012    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 01:16:18.296279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:18.296109    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 01:16:18.404102 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:16:18.404064    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:16:18.621736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:18.621645    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:19.296933 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:19.296897    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 01:16:19.297104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:19.296966    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 01:16:20.255086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.255048    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:20.303372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.303336    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 01:16:20.303636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.303611    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 01:16:20.622369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.622279    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"]
Oct 30 01:16:20.622855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.622842    4576 cpu_manager.go:414] "containerMap: removing container" podUID="01d77e55-3c6b-4ec9-a01b-8c5df878245d" containerName="revoke-keys"
Oct 30 01:16:20.622911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.622857    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d77e55-3c6b-4ec9-a01b-8c5df878245d" containerName="revoke-keys"
Oct 30 01:16:20.622994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.622985    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d77e55-3c6b-4ec9-a01b-8c5df878245d" containerName="revoke-keys"
Oct 30 01:16:20.626760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.626737    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.642830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.642800    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:20.645749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.645722    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"]
Oct 30 01:16:20.653483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.653457    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:16:20.730247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.730195    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5nd\" (UniqueName: \"kubernetes.io/projected/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-kube-api-access-hb5nd\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.730409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.730382    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.730518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.730496    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.757108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.757077    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"]
Oct 30 01:16:20.761552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.761530    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.788018 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.787989    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-d0ae0d-openshift-default-dockercfg-2cc5d\""
Oct 30 01:16:20.788222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.788023    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"]
Oct 30 01:16:20.832335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832302    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.832564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832346    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.832564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832405    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.832564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832447    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.832564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832540    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.832806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832584    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427d671-d974-43ef-a4d8-b255bab398e1-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.832806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832618    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427d671-d974-43ef-a4d8-b255bab398e1-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.832806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832657    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5nd\" (UniqueName: \"kubernetes.io/projected/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-kube-api-access-hb5nd\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.832806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832706    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.832806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832739    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzss2\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-kube-api-access-nzss2\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.832806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832789    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.833109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.832818    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.833169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.833118    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.835526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.835502    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.855859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.855825    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5nd\" (UniqueName: \"kubernetes.io/projected/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-kube-api-access-hb5nd\") pod \"maas-api-revoke-sub-464b3050-nt84v\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.934012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.933923    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.933971    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzss2\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-kube-api-access-nzss2\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934111    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934177    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934423    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934472    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934492    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-certs\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934575    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-data\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934589    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934680    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934717    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427d671-d974-43ef-a4d8-b255bab398e1-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934779    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-credential-socket\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.934899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.934818    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427d671-d974-43ef-a4d8-b255bab398e1-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.935562 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.935538    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427d671-d974-43ef-a4d8-b255bab398e1-istiod-ca-cert\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.937071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.937050    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-envoy\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.937740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.937722    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427d671-d974-43ef-a4d8-b255bab398e1-istio-podinfo\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.944697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.944674    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:20.956086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.956058    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-istio-token\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:20.956560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:20.956536    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzss2\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-kube-api-access-nzss2\") pod \"e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") " pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:21.076684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.076653    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:21.263599 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:21.263561    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72f61cc_5b6a_4ef2_91d6_e83e73ff1bcb.slice/crio-6e537a86832cd080ac691bc35285ddfa535cd8fdec5a87137c68813dec782c47 WatchSource:0}: Error finding container 6e537a86832cd080ac691bc35285ddfa535cd8fdec5a87137c68813dec782c47: Status 404 returned error can't find the container with id 6e537a86832cd080ac691bc35285ddfa535cd8fdec5a87137c68813dec782c47
Oct 30 01:16:21.264634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.264586    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"]
Oct 30 01:16:21.391920 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.391879    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:21.393028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.393001    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:21.393929 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.393906    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:16:21.408390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.408363    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"]
Oct 30 01:16:21.408856 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:21.408832    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6427d671_d974_43ef_a4d8_b255bab398e1.slice/crio-0bef313924b6ec1aeab1f35e54127baa4feb2e6634fee3732583b79a4714b497 WatchSource:0}: Error finding container 0bef313924b6ec1aeab1f35e54127baa4feb2e6634fee3732583b79a4714b497: Status 404 returned error can't find the container with id 0bef313924b6ec1aeab1f35e54127baa4feb2e6634fee3732583b79a4714b497
Oct 30 01:16:21.415193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.412207    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:21.415193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.412390    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:21.415193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:21.412480    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:16:22.000534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.000498    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:22.034289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.034251    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"]
Oct 30 01:16:22.039410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.039381    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"]
Oct 30 01:16:22.047292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.047190    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5d2ba8d-cnkt4"]
Oct 30 01:16:22.052347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.052313    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c30fdad-qrcmv"]
Oct 30 01:16:22.075151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.075115    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v" event={"ID":"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb","Type":"ContainerStarted","Data":"9359f9572e1380283b66b1267102a51282de27ab948f29d4b8045a0cb64d5167"}
Oct 30 01:16:22.075349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.075212    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v" event={"ID":"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb","Type":"ContainerStarted","Data":"6e537a86832cd080ac691bc35285ddfa535cd8fdec5a87137c68813dec782c47"}
Oct 30 01:16:22.077227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.077197    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" event={"ID":"6427d671-d974-43ef-a4d8-b255bab398e1","Type":"ContainerStarted","Data":"6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a"}
Oct 30 01:16:22.077389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.077295    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" event={"ID":"6427d671-d974-43ef-a4d8-b255bab398e1","Type":"ContainerStarted","Data":"0bef313924b6ec1aeab1f35e54127baa4feb2e6634fee3732583b79a4714b497"}
Oct 30 01:16:22.108775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.108747    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1416378-ba86-4a58-b135-04644376b588" path="/var/lib/kubelet/pods/f1416378-ba86-4a58-b135-04644376b588/volumes"
Oct 30 01:16:22.109427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.109400    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b9c3dc-6f68-4ebc-9632-b920bcd50b09" path="/var/lib/kubelet/pods/f9b9c3dc-6f68-4ebc-9632-b920bcd50b09/volumes"
Oct 30 01:16:22.121369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.121321    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v" podStartSLOduration=2.121306343 podStartE2EDuration="2.121306343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:22.120382973 +0000 UTC m=+2262.191706810" watchObservedRunningTime="2026-10-30 01:16:22.121306343 +0000 UTC m=+2262.192630177"
Oct 30 01:16:22.173484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:22.173432    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" podStartSLOduration=2.173404761 podStartE2EDuration="2.173404761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:22.169855459 +0000 UTC m=+2262.241179320" watchObservedRunningTime="2026-10-30 01:16:22.173404761 +0000 UTC m=+2262.244728596"
Oct 30 01:16:23.081252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:23.081201    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:23.087248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:23.087196    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:16:23.087498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:23.087471    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 01:16:23.106642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:23.106610    4576 generic.go:356] "Generic (PLEG): container finished" podID="a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb" containerID="9359f9572e1380283b66b1267102a51282de27ab948f29d4b8045a0cb64d5167" exitCode=0
Oct 30 01:16:23.107082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:23.107052    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v" event={"ID":"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb","Type":"ContainerDied","Data":"9359f9572e1380283b66b1267102a51282de27ab948f29d4b8045a0cb64d5167"}
Oct 30 01:16:23.595259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:23.589211    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:24.092413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.092368    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:16:24.092944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.092438    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 01:16:24.327327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.327304    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:24.418812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.418716    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-kube-api-access-hb5nd\" (UniqueName: \"kubernetes.io/projected/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-kube-api-access-hb5nd\") pod \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") "
Oct 30 01:16:24.418993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.418928    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-openshift-ca\") pod \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") "
Oct 30 01:16:24.419066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.419004    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-xks-ca\") pod \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\" (UID: \"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb\") "
Oct 30 01:16:24.419397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.419364    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-openshift-ca" pod "a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb" (UID: "a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:24.419739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.419647    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.422390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.422359    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-xks-ca" pod "a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb" (UID: "a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:24.422390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.422369    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-kube-api-access-hb5nd" pod "a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb" (UID: "a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb"). InnerVolumeSpecName "kube-api-access-hb5nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:24.520629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.520600    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5nd\" (UniqueName: \"kubernetes.io/projected/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-kube-api-access-hb5nd\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.520629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:24.520625    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:25.089261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.087139    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:16:25.089261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.087200    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 01:16:25.125571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.125533    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"
Oct 30 01:16:25.125954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.125570    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v" event={"ID":"a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb","Type":"ContainerDied","Data":"6e537a86832cd080ac691bc35285ddfa535cd8fdec5a87137c68813dec782c47"}
Oct 30 01:16:25.125954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.125625    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e537a86832cd080ac691bc35285ddfa535cd8fdec5a87137c68813dec782c47"
Oct 30 01:16:25.209209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.209168    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:25.647533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.645575    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"]
Oct 30 01:16:25.716529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.714946    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:25.895261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.892915    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"]
Oct 30 01:16:25.895261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.894020    4576 cpu_manager.go:414] "containerMap: removing container" podUID="a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb" containerName="revoke-keys"
Oct 30 01:16:25.895261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.894055    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb" containerName="revoke-keys"
Oct 30 01:16:25.895261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.894279    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb" containerName="revoke-keys"
Oct 30 01:16:25.913688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.913523    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:25.935448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.933375    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:16:25.935448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.934054    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:25.941709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:25.941604    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"]
Oct 30 01:16:26.053353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.053314    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ph4\" (UniqueName: \"kubernetes.io/projected/e03af931-e6a0-4f29-8c9f-d173ab579435-kube-api-access-w6ph4\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.053535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.053409    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.053535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.053527    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.078607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.078567    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:16:26.078882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.078852    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 01:16:26.155251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.155202    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.155732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.155381    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ph4\" (UniqueName: \"kubernetes.io/projected/e03af931-e6a0-4f29-8c9f-d173ab579435-kube-api-access-w6ph4\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.155732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.155413    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.156071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.156050    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.157778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.157758    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.189266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.188488    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ph4\" (UniqueName: \"kubernetes.io/projected/e03af931-e6a0-4f29-8c9f-d173ab579435-kube-api-access-w6ph4\") pod \"maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.251603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.251568    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:26.606633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:26.606601    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"]
Oct 30 01:16:26.607753 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:26.607727    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03af931_e6a0_4f29_8c9f_d173ab579435.slice/crio-9a5f9ba17d98885bbd91c7243b6e03598faf1c6691a159272a91bc04c91a07a0 WatchSource:0}: Error finding container 9a5f9ba17d98885bbd91c7243b6e03598faf1c6691a159272a91bc04c91a07a0: Status 404 returned error can't find the container with id 9a5f9ba17d98885bbd91c7243b6e03598faf1c6691a159272a91bc04c91a07a0
Oct 30 01:16:27.007167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:27.007140    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:27.101382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:27.100314    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:16:27.101382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:27.100371    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 01:16:27.148290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:27.148212    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk" event={"ID":"e03af931-e6a0-4f29-8c9f-d173ab579435","Type":"ContainerStarted","Data":"5710fa627c8543ddcacf684bae9d6b11bae9610056adc28fc26f57db0ad65d28"}
Oct 30 01:16:27.148624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:27.148594    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk" event={"ID":"e03af931-e6a0-4f29-8c9f-d173ab579435","Type":"ContainerStarted","Data":"9a5f9ba17d98885bbd91c7243b6e03598faf1c6691a159272a91bc04c91a07a0"}
Oct 30 01:16:27.191514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:27.191451    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk" podStartSLOduration=2.191430072 podStartE2EDuration="2.191430072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:27.189429304 +0000 UTC m=+2267.260753142" watchObservedRunningTime="2026-10-30 01:16:27.191430072 +0000 UTC m=+2267.262753909"
Oct 30 01:16:28.077975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:28.077940    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 01:16:28.078176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:28.078015    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 01:16:28.158363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:28.158325    4576 generic.go:356] "Generic (PLEG): container finished" podID="e03af931-e6a0-4f29-8c9f-d173ab579435" containerID="5710fa627c8543ddcacf684bae9d6b11bae9610056adc28fc26f57db0ad65d28" exitCode=0
Oct 30 01:16:28.158528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:28.158408    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk" event={"ID":"e03af931-e6a0-4f29-8c9f-d173ab579435","Type":"ContainerDied","Data":"5710fa627c8543ddcacf684bae9d6b11bae9610056adc28fc26f57db0ad65d28"}
Oct 30 01:16:28.576935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:28.576895    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:16:28.745214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:28.745176    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:29.082062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.082032    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:29.082509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.082479    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:29.083348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.083329    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:16:29.328143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.328121    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:29.438446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.438407    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-xks-ca\") pod \"e03af931-e6a0-4f29-8c9f-d173ab579435\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") "
Oct 30 01:16:29.438667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.438647    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03af931-e6a0-4f29-8c9f-d173ab579435-kube-api-access-w6ph4\" (UniqueName: \"kubernetes.io/projected/e03af931-e6a0-4f29-8c9f-d173ab579435-kube-api-access-w6ph4\") pod \"e03af931-e6a0-4f29-8c9f-d173ab579435\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") "
Oct 30 01:16:29.438806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.438775    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-openshift-ca\") pod \"e03af931-e6a0-4f29-8c9f-d173ab579435\" (UID: \"e03af931-e6a0-4f29-8c9f-d173ab579435\") "
Oct 30 01:16:29.439211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.439182    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-openshift-ca" pod "e03af931-e6a0-4f29-8c9f-d173ab579435" (UID: "e03af931-e6a0-4f29-8c9f-d173ab579435"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:29.439438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.439415    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:29.441304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.441274    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-xks-ca" pod "e03af931-e6a0-4f29-8c9f-d173ab579435" (UID: "e03af931-e6a0-4f29-8c9f-d173ab579435"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:29.441595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.441569    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03af931-e6a0-4f29-8c9f-d173ab579435-kube-api-access-w6ph4" pod "e03af931-e6a0-4f29-8c9f-d173ab579435" (UID: "e03af931-e6a0-4f29-8c9f-d173ab579435"). InnerVolumeSpecName "kube-api-access-w6ph4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:29.540676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.540642    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6ph4\" (UniqueName: \"kubernetes.io/projected/e03af931-e6a0-4f29-8c9f-d173ab579435-kube-api-access-w6ph4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:29.540676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:29.540672    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03af931-e6a0-4f29-8c9f-d173ab579435-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:30.189010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:30.187993    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"
Oct 30 01:16:30.189010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:30.188607    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk" event={"ID":"e03af931-e6a0-4f29-8c9f-d173ab579435","Type":"ContainerDied","Data":"9a5f9ba17d98885bbd91c7243b6e03598faf1c6691a159272a91bc04c91a07a0"}
Oct 30 01:16:30.189010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:30.188659    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a5f9ba17d98885bbd91c7243b6e03598faf1c6691a159272a91bc04c91a07a0"
Oct 30 01:16:31.177977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.177944    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:31.287671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.287634    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"]
Oct 30 01:16:31.288387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.288369    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e03af931-e6a0-4f29-8c9f-d173ab579435" containerName="revoke-keys"
Oct 30 01:16:31.288387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.288388    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03af931-e6a0-4f29-8c9f-d173ab579435" containerName="revoke-keys"
Oct 30 01:16:31.288571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.288560    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03af931-e6a0-4f29-8c9f-d173ab579435" containerName="revoke-keys"
Oct 30 01:16:31.292611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.292583    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.313498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.313470    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"]
Oct 30 01:16:31.316620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.316455    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:31.316759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.316622    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:16:31.395148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.395117    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.395362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.395287    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.395478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.395465    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsm9\" (UniqueName: \"kubernetes.io/projected/cc37556a-5835-4b59-9e74-a4034961304a-kube-api-access-vrsm9\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.497048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.497009    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.497271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.497226    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsm9\" (UniqueName: \"kubernetes.io/projected/cc37556a-5835-4b59-9e74-a4034961304a-kube-api-access-vrsm9\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.497334 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.497291    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.497795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.497768    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.499765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.499739    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.518371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.518327    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsm9\" (UniqueName: \"kubernetes.io/projected/cc37556a-5835-4b59-9e74-a4034961304a-kube-api-access-vrsm9\") pod \"maas-api-revoke-sub-4d0afe36-swtj6\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.625389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.625354    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:31.929315 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:31.929287    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc37556a_5835_4b59_9e74_a4034961304a.slice/crio-0c50432ce15a2ff04f4a084f14a7738ccbd315f1fa28ddf6aedb2feab3bf8384 WatchSource:0}: Error finding container 0c50432ce15a2ff04f4a084f14a7738ccbd315f1fa28ddf6aedb2feab3bf8384: Status 404 returned error can't find the container with id 0c50432ce15a2ff04f4a084f14a7738ccbd315f1fa28ddf6aedb2feab3bf8384
Oct 30 01:16:31.930268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:31.930222    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"]
Oct 30 01:16:32.202139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:32.202092    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6" event={"ID":"cc37556a-5835-4b59-9e74-a4034961304a","Type":"ContainerStarted","Data":"0c50432ce15a2ff04f4a084f14a7738ccbd315f1fa28ddf6aedb2feab3bf8384"}
Oct 30 01:16:32.969372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:32.969343    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:33.030992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.030928    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"]
Oct 30 01:16:33.043431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.043400    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a91efc1-7x56g"]
Oct 30 01:16:33.212425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.212388    4576 generic.go:356] "Generic (PLEG): container finished" podID="cc37556a-5835-4b59-9e74-a4034961304a" containerID="f3d0ea39a965c618b2c8f49ee1118f628e63c59f4facc7e270e15d693df99f04" exitCode=0
Oct 30 01:16:33.212603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.212439    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6" event={"ID":"cc37556a-5835-4b59-9e74-a4034961304a","Type":"ContainerDied","Data":"f3d0ea39a965c618b2c8f49ee1118f628e63c59f4facc7e270e15d693df99f04"}
Oct 30 01:16:33.305036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.305004    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"]
Oct 30 01:16:33.309204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.309184    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.326747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.326720    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"]
Oct 30 01:16:33.425223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.425189    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.425563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.425540    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.425638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.425618    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzps\" (UniqueName: \"kubernetes.io/projected/8e320070-870c-4d31-9a84-f4981f106f1d-kube-api-access-6nzps\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.527505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.527418    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.527505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.527491    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzps\" (UniqueName: \"kubernetes.io/projected/8e320070-870c-4d31-9a84-f4981f106f1d-kube-api-access-6nzps\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.527736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.527540    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.528040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.528020    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.530064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.530043    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.548950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.548918    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzps\" (UniqueName: \"kubernetes.io/projected/8e320070-870c-4d31-9a84-f4981f106f1d-kube-api-access-6nzps\") pod \"maas-api-revoke-sub-0179dbb6-2mdhw\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.624392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.624361    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:33.942050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:33.942018    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"]
Oct 30 01:16:33.943302 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:33.943275    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e320070_870c_4d31_9a84_f4981f106f1d.slice/crio-35c052ae02cd03c446b54d48d277843d5df93568938190176a4d0e6d2bc6ac06 WatchSource:0}: Error finding container 35c052ae02cd03c446b54d48d277843d5df93568938190176a4d0e6d2bc6ac06: Status 404 returned error can't find the container with id 35c052ae02cd03c446b54d48d277843d5df93568938190176a4d0e6d2bc6ac06
Oct 30 01:16:34.109275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.109221    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c3ae00-3ded-496c-86d4-b4bc3c9917a9" path="/var/lib/kubelet/pods/69c3ae00-3ded-496c-86d4-b4bc3c9917a9/volumes"
Oct 30 01:16:34.221141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.221069    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw" event={"ID":"8e320070-870c-4d31-9a84-f4981f106f1d","Type":"ContainerStarted","Data":"35c052ae02cd03c446b54d48d277843d5df93568938190176a4d0e6d2bc6ac06"}
Oct 30 01:16:34.364818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.364792    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:34.440133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.440104    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-xks-ca\") pod \"cc37556a-5835-4b59-9e74-a4034961304a\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") "
Oct 30 01:16:34.440329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.440314    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-openshift-ca\") pod \"cc37556a-5835-4b59-9e74-a4034961304a\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") "
Oct 30 01:16:34.440404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.440354    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc37556a-5835-4b59-9e74-a4034961304a-kube-api-access-vrsm9\" (UniqueName: \"kubernetes.io/projected/cc37556a-5835-4b59-9e74-a4034961304a-kube-api-access-vrsm9\") pod \"cc37556a-5835-4b59-9e74-a4034961304a\" (UID: \"cc37556a-5835-4b59-9e74-a4034961304a\") "
Oct 30 01:16:34.440727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.440699    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-openshift-ca" pod "cc37556a-5835-4b59-9e74-a4034961304a" (UID: "cc37556a-5835-4b59-9e74-a4034961304a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:34.441019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.440995    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:34.442757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.442731    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc37556a-5835-4b59-9e74-a4034961304a-kube-api-access-vrsm9" pod "cc37556a-5835-4b59-9e74-a4034961304a" (UID: "cc37556a-5835-4b59-9e74-a4034961304a"). InnerVolumeSpecName "kube-api-access-vrsm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:34.442757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.442731    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-xks-ca" pod "cc37556a-5835-4b59-9e74-a4034961304a" (UID: "cc37556a-5835-4b59-9e74-a4034961304a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:34.541968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.541894    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrsm9\" (UniqueName: \"kubernetes.io/projected/cc37556a-5835-4b59-9e74-a4034961304a-kube-api-access-vrsm9\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:34.541968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.541927    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc37556a-5835-4b59-9e74-a4034961304a-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:34.560119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:34.560091    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:35.230745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:35.230712    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"
Oct 30 01:16:35.231195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:35.230744    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6" event={"ID":"cc37556a-5835-4b59-9e74-a4034961304a","Type":"ContainerDied","Data":"0c50432ce15a2ff04f4a084f14a7738ccbd315f1fa28ddf6aedb2feab3bf8384"}
Oct 30 01:16:35.231195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:35.230792    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c50432ce15a2ff04f4a084f14a7738ccbd315f1fa28ddf6aedb2feab3bf8384"
Oct 30 01:16:35.232591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:35.232565    4576 generic.go:356] "Generic (PLEG): container finished" podID="8e320070-870c-4d31-9a84-f4981f106f1d" containerID="9ff591e6424e5d06167c511a5d214fd13052a2d2c51c7bdbefb1ca206cf51cdb" exitCode=0
Oct 30 01:16:35.232721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:35.232633    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw" event={"ID":"8e320070-870c-4d31-9a84-f4981f106f1d","Type":"ContainerDied","Data":"9ff591e6424e5d06167c511a5d214fd13052a2d2c51c7bdbefb1ca206cf51cdb"}
Oct 30 01:16:36.009577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.009541    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:36.122838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.122796    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:36.380072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.380042    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"]
Oct 30 01:16:36.412303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.412275    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:36.470057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.470028    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-xks-ca\") pod \"8e320070-870c-4d31-9a84-f4981f106f1d\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") "
Oct 30 01:16:36.470302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.470198    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-openshift-ca\") pod \"8e320070-870c-4d31-9a84-f4981f106f1d\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") "
Oct 30 01:16:36.470395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.470315    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e320070-870c-4d31-9a84-f4981f106f1d-kube-api-access-6nzps\" (UniqueName: \"kubernetes.io/projected/8e320070-870c-4d31-9a84-f4981f106f1d-kube-api-access-6nzps\") pod \"8e320070-870c-4d31-9a84-f4981f106f1d\" (UID: \"8e320070-870c-4d31-9a84-f4981f106f1d\") "
Oct 30 01:16:36.471818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.471791    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-openshift-ca" pod "8e320070-870c-4d31-9a84-f4981f106f1d" (UID: "8e320070-870c-4d31-9a84-f4981f106f1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:36.475124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.475095    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e320070-870c-4d31-9a84-f4981f106f1d-kube-api-access-6nzps" pod "8e320070-870c-4d31-9a84-f4981f106f1d" (UID: "8e320070-870c-4d31-9a84-f4981f106f1d"). InnerVolumeSpecName "kube-api-access-6nzps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:36.479760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.479706    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-xks-ca" pod "8e320070-870c-4d31-9a84-f4981f106f1d" (UID: "8e320070-870c-4d31-9a84-f4981f106f1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:36.502289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.501406    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:36.572496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.572395    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.572496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.572439    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e320070-870c-4d31-9a84-f4981f106f1d-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.572496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.572453    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nzps\" (UniqueName: \"kubernetes.io/projected/8e320070-870c-4d31-9a84-f4981f106f1d-kube-api-access-6nzps\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:36.625996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:36.625944    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:37.250010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:37.249977    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"
Oct 30 01:16:37.250183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:37.250004    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw" event={"ID":"8e320070-870c-4d31-9a84-f4981f106f1d","Type":"ContainerDied","Data":"35c052ae02cd03c446b54d48d277843d5df93568938190176a4d0e6d2bc6ac06"}
Oct 30 01:16:37.250183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:37.250063    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c052ae02cd03c446b54d48d277843d5df93568938190176a4d0e6d2bc6ac06"
Oct 30 01:16:37.769817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:37.769784    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:39.100612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:39.100567    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:39.255988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:39.255953    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:39.572612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:39.572575    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:40.716670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:40.716627    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd" podUID="f981d537-1da3-417f-b726-b1d04056884a" containerName="main" containerID="cri-o://a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012" gracePeriod=45
Oct 30 01:16:40.871712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:40.871677    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:41.002443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.000584    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:41.029253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.028994    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:16:41.115127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.115086    4576 scope.go:122] "RemoveContainer" containerID="625643df1db62418f62b5624cb98570f5786360505c88c9d056ba6d47519971b"
Oct 30 01:16:41.132906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.132879    4576 scope.go:122] "RemoveContainer" containerID="d66f6488e4f597f931ce36237022c31be460158f9fbbc20d81fe5708674fbf05"
Oct 30 01:16:41.148723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.148699    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f981d537-1da3-417f-b726-b1d04056884a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f981d537-1da3-417f-b726-b1d04056884a-tls-certs\") pod \"f981d537-1da3-417f-b726-b1d04056884a\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") "
Oct 30 01:16:41.148859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.148790    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-tmp-dir\") pod \"f981d537-1da3-417f-b726-b1d04056884a\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") "
Oct 30 01:16:41.148859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.148824    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-model-cache\") pod \"f981d537-1da3-417f-b726-b1d04056884a\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") "
Oct 30 01:16:41.148981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.148875    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-dshm\") pod \"f981d537-1da3-417f-b726-b1d04056884a\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") "
Oct 30 01:16:41.149161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.149018    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-home\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-home\") pod \"f981d537-1da3-417f-b726-b1d04056884a\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") "
Oct 30 01:16:41.149161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.149108    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f981d537-1da3-417f-b726-b1d04056884a-kube-api-access-c2hkv\" (UniqueName: \"kubernetes.io/projected/f981d537-1da3-417f-b726-b1d04056884a-kube-api-access-c2hkv\") pod \"f981d537-1da3-417f-b726-b1d04056884a\" (UID: \"f981d537-1da3-417f-b726-b1d04056884a\") "
Oct 30 01:16:41.149161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.149110    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-model-cache" pod "f981d537-1da3-417f-b726-b1d04056884a" (UID: "f981d537-1da3-417f-b726-b1d04056884a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:41.149161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.149119    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-tmp-dir" pod "f981d537-1da3-417f-b726-b1d04056884a" (UID: "f981d537-1da3-417f-b726-b1d04056884a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:41.149487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.149368    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-home" pod "f981d537-1da3-417f-b726-b1d04056884a" (UID: "f981d537-1da3-417f-b726-b1d04056884a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:41.150109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.150083    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:41.150289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.150115    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:41.150289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.150131    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:41.151523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.151498    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-dshm" pod "f981d537-1da3-417f-b726-b1d04056884a" (UID: "f981d537-1da3-417f-b726-b1d04056884a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:41.151779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.151759    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f981d537-1da3-417f-b726-b1d04056884a-tls-certs" pod "f981d537-1da3-417f-b726-b1d04056884a" (UID: "f981d537-1da3-417f-b726-b1d04056884a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:41.151888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.151780    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f981d537-1da3-417f-b726-b1d04056884a-kube-api-access-c2hkv" pod "f981d537-1da3-417f-b726-b1d04056884a" (UID: "f981d537-1da3-417f-b726-b1d04056884a"). InnerVolumeSpecName "kube-api-access-c2hkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:41.252037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.251947    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2hkv\" (UniqueName: \"kubernetes.io/projected/f981d537-1da3-417f-b726-b1d04056884a-kube-api-access-c2hkv\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:41.252037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.251980    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f981d537-1da3-417f-b726-b1d04056884a-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:41.252037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.251993    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f981d537-1da3-417f-b726-b1d04056884a-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:41.287927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.287890    4576 generic.go:356] "Generic (PLEG): container finished" podID="f981d537-1da3-417f-b726-b1d04056884a" containerID="a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012" exitCode=0
Oct 30 01:16:41.288102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.287938    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"
Oct 30 01:16:41.288102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.287966    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd" event={"ID":"f981d537-1da3-417f-b726-b1d04056884a","Type":"ContainerDied","Data":"a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012"}
Oct 30 01:16:41.288102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.288027    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd" event={"ID":"f981d537-1da3-417f-b726-b1d04056884a","Type":"ContainerDied","Data":"ea607f94fb95c8443b57922d83c3e9117380f97ff9939aa89392ff7da9ac66b6"}
Oct 30 01:16:41.288102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.288053    4576 scope.go:122] "RemoveContainer" containerID="a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012"
Oct 30 01:16:41.302097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.302071    4576 scope.go:122] "RemoveContainer" containerID="a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012"
Oct 30 01:16:41.302453 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:16:41.302421    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012\": container with ID starting with a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012 not found: ID does not exist" containerID="a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012"
Oct 30 01:16:41.302593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.302461    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012"} err="failed to get container status \"a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012\": rpc error: code = NotFound desc = could not find container \"a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012\": container with ID starting with a641dea26372cb8a87484138dc1193f500d191dafd3462d2cd78c118104ea012 not found: ID does not exist"
Oct 30 01:16:41.348058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.348031    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"]
Oct 30 01:16:41.355479 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.355118    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-95fe33/test-model-95fe33-kserve-544ccdf65-twrwd"]
Oct 30 01:16:41.380854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.380788    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l" podUID="e6576139-2679-4a30-b6dd-ec97d4dd6a41" containerName="payload-pre-processing" containerID="cri-o://aba30fa127335f5f4dd65447711ffb1d151024d65a5d170c06a2ac6328440061" gracePeriod=25
Oct 30 01:16:41.856406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:41.856368    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:42.045260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:42.042805    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:42.115312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:42.115197    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f981d537-1da3-417f-b726-b1d04056884a" path="/var/lib/kubelet/pods/f981d537-1da3-417f-b726-b1d04056884a/volumes"
Oct 30 01:16:42.624337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:42.624300    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:43.602530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.602498    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"]
Oct 30 01:16:43.603093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603079    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8e320070-870c-4d31-9a84-f4981f106f1d" containerName="revoke-keys"
Oct 30 01:16:43.603141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603093    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e320070-870c-4d31-9a84-f4981f106f1d" containerName="revoke-keys"
Oct 30 01:16:43.603141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603130    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f981d537-1da3-417f-b726-b1d04056884a" containerName="main"
Oct 30 01:16:43.603141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603135    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f981d537-1da3-417f-b726-b1d04056884a" containerName="main"
Oct 30 01:16:43.603253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603152    4576 cpu_manager.go:414] "containerMap: removing container" podUID="cc37556a-5835-4b59-9e74-a4034961304a" containerName="revoke-keys"
Oct 30 01:16:43.603253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603157    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc37556a-5835-4b59-9e74-a4034961304a" containerName="revoke-keys"
Oct 30 01:16:43.603323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603265    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f981d537-1da3-417f-b726-b1d04056884a" containerName="main"
Oct 30 01:16:43.603323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603273    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc37556a-5835-4b59-9e74-a4034961304a" containerName="revoke-keys"
Oct 30 01:16:43.603323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.603285    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e320070-870c-4d31-9a84-f4981f106f1d" containerName="revoke-keys"
Oct 30 01:16:43.608850 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.608828    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.623482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.623454    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"]
Oct 30 01:16:43.632036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.631863    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:43.632036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.631973    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:16:43.690529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.690495    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.690529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.690535    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrw4v\" (UniqueName: \"kubernetes.io/projected/7b90aab1-a761-420c-a30e-b503402360b7-kube-api-access-xrw4v\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.690762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.690566    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.792687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.792657    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.792892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.792693    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrw4v\" (UniqueName: \"kubernetes.io/projected/7b90aab1-a761-420c-a30e-b503402360b7-kube-api-access-xrw4v\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.792892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.792723    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.793379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.793357    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.795202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.795179    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.814978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.814949    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrw4v\" (UniqueName: \"kubernetes.io/projected/7b90aab1-a761-420c-a30e-b503402360b7-kube-api-access-xrw4v\") pod \"maas-api-revoke-sub-a8866816-bxgmj\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:43.933879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:43.933805    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:44.197456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:44.197414    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:44.248838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:44.248800    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"]
Oct 30 01:16:44.315267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:44.315222    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj" event={"ID":"7b90aab1-a761-420c-a30e-b503402360b7","Type":"ContainerStarted","Data":"8c29b734342b70ff426eb7253cfd32b2166ede1ecb711eae3a5628e6adc93889"}
Oct 30 01:16:45.324036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:45.324002    4576 generic.go:356] "Generic (PLEG): container finished" podID="7b90aab1-a761-420c-a30e-b503402360b7" containerID="a2d3ceeed105c9df35f4376f505be281fce2d5405ccb5f1b9ca1cb6185d29a8e" exitCode=0
Oct 30 01:16:45.324487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:45.324066    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj" event={"ID":"7b90aab1-a761-420c-a30e-b503402360b7","Type":"ContainerDied","Data":"a2d3ceeed105c9df35f4376f505be281fce2d5405ccb5f1b9ca1cb6185d29a8e"}
Oct 30 01:16:45.752655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:45.752617    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:46.506127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.506093    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:46.632677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.632593    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-openshift-ca\") pod \"7b90aab1-a761-420c-a30e-b503402360b7\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") "
Oct 30 01:16:46.632879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.632678    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b90aab1-a761-420c-a30e-b503402360b7-kube-api-access-xrw4v\" (UniqueName: \"kubernetes.io/projected/7b90aab1-a761-420c-a30e-b503402360b7-kube-api-access-xrw4v\") pod \"7b90aab1-a761-420c-a30e-b503402360b7\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") "
Oct 30 01:16:46.632879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.632726    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-xks-ca\") pod \"7b90aab1-a761-420c-a30e-b503402360b7\" (UID: \"7b90aab1-a761-420c-a30e-b503402360b7\") "
Oct 30 01:16:46.632998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.632955    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-openshift-ca" pod "7b90aab1-a761-420c-a30e-b503402360b7" (UID: "7b90aab1-a761-420c-a30e-b503402360b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:46.633406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.633377    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:46.635107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.635068    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-xks-ca" pod "7b90aab1-a761-420c-a30e-b503402360b7" (UID: "7b90aab1-a761-420c-a30e-b503402360b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:46.635206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.635112    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b90aab1-a761-420c-a30e-b503402360b7-kube-api-access-xrw4v" pod "7b90aab1-a761-420c-a30e-b503402360b7" (UID: "7b90aab1-a761-420c-a30e-b503402360b7"). InnerVolumeSpecName "kube-api-access-xrw4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:46.734388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.734349    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrw4v\" (UniqueName: \"kubernetes.io/projected/7b90aab1-a761-420c-a30e-b503402360b7-kube-api-access-xrw4v\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:46.734388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:46.734381    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b90aab1-a761-420c-a30e-b503402360b7-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:16:47.228118 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:47.228080    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:47.342058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:47.342030    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"
Oct 30 01:16:47.342058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:47.342048    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj" event={"ID":"7b90aab1-a761-420c-a30e-b503402360b7","Type":"ContainerDied","Data":"8c29b734342b70ff426eb7253cfd32b2166ede1ecb711eae3a5628e6adc93889"}
Oct 30 01:16:47.342292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:47.342096    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c29b734342b70ff426eb7253cfd32b2166ede1ecb711eae3a5628e6adc93889"
Oct 30 01:16:48.234527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:48.234492    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l" podUID="e6576139-2679-4a30-b6dd-ec97d4dd6a41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:16:48.381737 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:16:48.381703    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:16:48.823121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:48.823077    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:48.935025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:48.934935    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:49.041779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:49.041745    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:50.212746 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:50.212701    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:50.818825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:50.818786    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:50.948802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:50.948765    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:50.952193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:50.952169    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"]
Oct 30 01:16:50.963457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:50.963432    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b312f7-2slwk"]
Oct 30 01:16:51.404850 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:51.404773    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:52.109453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:52.109422    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03af931-e6a0-4f29-8c9f-d173ab579435" path="/var/lib/kubelet/pods/e03af931-e6a0-4f29-8c9f-d173ab579435/volumes"
Oct 30 01:16:52.764372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:52.764338    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:53.025629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:53.025542    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"]
Oct 30 01:16:53.039978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:53.039945    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1e1fee-zzcqr"]
Oct 30 01:16:54.109792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:54.109747    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c36dac6-0b58-4af3-856a-3f7cea301cde" path="/var/lib/kubelet/pods/2c36dac6-0b58-4af3-856a-3f7cea301cde/volumes"
Oct 30 01:16:54.255261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:54.255199    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:55.687984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:55.687953    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:57.243814 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:57.243778    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:57.356770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:57.356736    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:57.625315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:57.625194    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"]
Oct 30 01:16:57.625565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:57.625535    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy" containerID="cri-o://44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c" gracePeriod=30
Oct 30 01:16:57.644216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:57.644187    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:58.090789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.090752    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"]
Oct 30 01:16:58.091392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.091377    4576 cpu_manager.go:414] "containerMap: removing container" podUID="7b90aab1-a761-420c-a30e-b503402360b7" containerName="revoke-keys"
Oct 30 01:16:58.091441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.091393    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b90aab1-a761-420c-a30e-b503402360b7" containerName="revoke-keys"
Oct 30 01:16:58.091508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.091499    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b90aab1-a761-420c-a30e-b503402360b7" containerName="revoke-keys"
Oct 30 01:16:58.096539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.096520    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.128091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.128061    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:16:58.128622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.128602    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"]
Oct 30 01:16:58.128794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.128778    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:16:58.219577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.219533    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.219807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.219591    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.219807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.219666    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86pm6\" (UniqueName: \"kubernetes.io/projected/12b72cbc-9268-4b5f-bece-5a90b34dec79-kube-api-access-86pm6\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.235032 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.235000    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l" podUID="e6576139-2679-4a30-b6dd-ec97d4dd6a41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:16:58.321569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.321526    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.321990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.321576    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.321990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.321630    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86pm6\" (UniqueName: \"kubernetes.io/projected/12b72cbc-9268-4b5f-bece-5a90b34dec79-kube-api-access-86pm6\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.322318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.322297    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.324119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.324095    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.342985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.342922    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86pm6\" (UniqueName: \"kubernetes.io/projected/12b72cbc-9268-4b5f-bece-5a90b34dec79-kube-api-access-86pm6\") pod \"maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.422301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.422266    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:16:58.732524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.732495    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"]
Oct 30 01:16:58.732831 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:16:58.732797    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b72cbc_9268_4b5f_bece_5a90b34dec79.slice/crio-bfe31a64e1c8e3e9ab7ca50d577e61d3adf7aef91a36aed857df8e2a9392d494 WatchSource:0}: Error finding container bfe31a64e1c8e3e9ab7ca50d577e61d3adf7aef91a36aed857df8e2a9392d494: Status 404 returned error can't find the container with id bfe31a64e1c8e3e9ab7ca50d577e61d3adf7aef91a36aed857df8e2a9392d494
Oct 30 01:16:58.866428 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:58.866392    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:59.273985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:59.273898    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:16:59.489261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:59.489024    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82" event={"ID":"12b72cbc-9268-4b5f-bece-5a90b34dec79","Type":"ContainerStarted","Data":"df872acd4fd8bd2db4e4a925016a666a6854ac8c058eba4fe89884e044ed9d07"}
Oct 30 01:16:59.489261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:59.489084    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82" event={"ID":"12b72cbc-9268-4b5f-bece-5a90b34dec79","Type":"ContainerStarted","Data":"bfe31a64e1c8e3e9ab7ca50d577e61d3adf7aef91a36aed857df8e2a9392d494"}
Oct 30 01:16:59.535804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:59.535693    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82" podStartSLOduration=1.535674164 podStartE2EDuration="1.535674164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:59.529453594 +0000 UTC m=+2299.600777430" watchObservedRunningTime="2026-10-30 01:16:59.535674164 +0000 UTC m=+2299.606997999"
Oct 30 01:16:59.652545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:59.652101    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"]
Oct 30 01:16:59.695576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:16:59.695540    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:00.498330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:00.498299    4576 generic.go:356] "Generic (PLEG): container finished" podID="12b72cbc-9268-4b5f-bece-5a90b34dec79" containerID="df872acd4fd8bd2db4e4a925016a666a6854ac8c058eba4fe89884e044ed9d07" exitCode=0
Oct 30 01:17:00.498842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:00.498386    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82" event={"ID":"12b72cbc-9268-4b5f-bece-5a90b34dec79","Type":"ContainerDied","Data":"df872acd4fd8bd2db4e4a925016a666a6854ac8c058eba4fe89884e044ed9d07"}
Oct 30 01:17:00.609492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:00.609458    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:01.506907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.506871    4576 generic.go:356] "Generic (PLEG): container finished" podID="e6576139-2679-4a30-b6dd-ec97d4dd6a41" containerID="aba30fa127335f5f4dd65447711ffb1d151024d65a5d170c06a2ac6328440061" exitCode=0
Oct 30 01:17:01.507274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.506942    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l" event={"ID":"e6576139-2679-4a30-b6dd-ec97d4dd6a41","Type":"ContainerDied","Data":"aba30fa127335f5f4dd65447711ffb1d151024d65a5d170c06a2ac6328440061"}
Oct 30 01:17:01.687823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.687794    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:17:01.721391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.721369    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:17:01.791425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.791397    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6576139-2679-4a30-b6dd-ec97d4dd6a41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6576139-2679-4a30-b6dd-ec97d4dd6a41-plugins-config-volume\") pod \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\" (UID: \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\") "
Oct 30 01:17:01.791611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.791598    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6576139-2679-4a30-b6dd-ec97d4dd6a41-kube-api-access-gk5g8\" (UniqueName: \"kubernetes.io/projected/e6576139-2679-4a30-b6dd-ec97d4dd6a41-kube-api-access-gk5g8\") pod \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\" (UID: \"e6576139-2679-4a30-b6dd-ec97d4dd6a41\") "
Oct 30 01:17:01.791877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.791846    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6576139-2679-4a30-b6dd-ec97d4dd6a41-plugins-config-volume" pod "e6576139-2679-4a30-b6dd-ec97d4dd6a41" (UID: "e6576139-2679-4a30-b6dd-ec97d4dd6a41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:01.792114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.792094    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6576139-2679-4a30-b6dd-ec97d4dd6a41-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:01.794103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.794070    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6576139-2679-4a30-b6dd-ec97d4dd6a41-kube-api-access-gk5g8" pod "e6576139-2679-4a30-b6dd-ec97d4dd6a41" (UID: "e6576139-2679-4a30-b6dd-ec97d4dd6a41"). InnerVolumeSpecName "kube-api-access-gk5g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:01.893190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.893159    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-xks-ca\") pod \"12b72cbc-9268-4b5f-bece-5a90b34dec79\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") "
Oct 30 01:17:01.893375 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.893271    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-openshift-ca\") pod \"12b72cbc-9268-4b5f-bece-5a90b34dec79\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") "
Oct 30 01:17:01.893447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.893430    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b72cbc-9268-4b5f-bece-5a90b34dec79-kube-api-access-86pm6\" (UniqueName: \"kubernetes.io/projected/12b72cbc-9268-4b5f-bece-5a90b34dec79-kube-api-access-86pm6\") pod \"12b72cbc-9268-4b5f-bece-5a90b34dec79\" (UID: \"12b72cbc-9268-4b5f-bece-5a90b34dec79\") "
Oct 30 01:17:01.893615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.893590    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-openshift-ca" pod "12b72cbc-9268-4b5f-bece-5a90b34dec79" (UID: "12b72cbc-9268-4b5f-bece-5a90b34dec79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:01.894015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.893984    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:01.894015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.894010    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5g8\" (UniqueName: \"kubernetes.io/projected/e6576139-2679-4a30-b6dd-ec97d4dd6a41-kube-api-access-gk5g8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:01.895391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.895369    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-xks-ca" pod "12b72cbc-9268-4b5f-bece-5a90b34dec79" (UID: "12b72cbc-9268-4b5f-bece-5a90b34dec79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:01.895773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.895749    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b72cbc-9268-4b5f-bece-5a90b34dec79-kube-api-access-86pm6" pod "12b72cbc-9268-4b5f-bece-5a90b34dec79" (UID: "12b72cbc-9268-4b5f-bece-5a90b34dec79"). InnerVolumeSpecName "kube-api-access-86pm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:01.994890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.994857    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86pm6\" (UniqueName: \"kubernetes.io/projected/12b72cbc-9268-4b5f-bece-5a90b34dec79-kube-api-access-86pm6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:01.994890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:01.994886    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b72cbc-9268-4b5f-bece-5a90b34dec79-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:02.260970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.260900    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:02.516461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.516379    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"
Oct 30 01:17:02.516461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.516414    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82" event={"ID":"12b72cbc-9268-4b5f-bece-5a90b34dec79","Type":"ContainerDied","Data":"bfe31a64e1c8e3e9ab7ca50d577e61d3adf7aef91a36aed857df8e2a9392d494"}
Oct 30 01:17:02.517139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.516465    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe31a64e1c8e3e9ab7ca50d577e61d3adf7aef91a36aed857df8e2a9392d494"
Oct 30 01:17:02.518295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.518275    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"
Oct 30 01:17:02.518418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.518267    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l" event={"ID":"e6576139-2679-4a30-b6dd-ec97d4dd6a41","Type":"ContainerDied","Data":"1eeff144b5196194a5e219b7b04531a305a0834ac0ac2b765e674ce37939e1c3"}
Oct 30 01:17:02.518418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.518387    4576 scope.go:122] "RemoveContainer" containerID="aba30fa127335f5f4dd65447711ffb1d151024d65a5d170c06a2ac6328440061"
Oct 30 01:17:02.649621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.649588    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"]
Oct 30 01:17:02.655977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.655953    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b312f7-fcdbc87b7-5jx4l"]
Oct 30 01:17:02.934656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:02.934625    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:17:03.111635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111552    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-kube-api-access-jz8lm\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-kube-api-access-jz8lm\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.111635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111609    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-podinfo\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.111870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111641    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-data\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.111870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111672    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-envoy\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.111870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111749    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-certs\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.111870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111780    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-credential-socket\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.112086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111871    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-socket\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.112086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111912    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4028e239-916a-4474-9da9-bb2f47dfb84d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028e239-916a-4474-9da9-bb2f47dfb84d-istiod-ca-cert\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.112086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111972    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-data" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:03.112086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.111982    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-token\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-token\") pod \"4028e239-916a-4474-9da9-bb2f47dfb84d\" (UID: \"4028e239-916a-4474-9da9-bb2f47dfb84d\") "
Oct 30 01:17:03.112317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.112164    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-credential-socket" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:03.112376 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.112347    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-certs" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:03.112840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.112798    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4028e239-916a-4474-9da9-bb2f47dfb84d-istiod-ca-cert" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:03.112840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.112816    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.112840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.112838    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.113063 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.112852    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.113397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.113376    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-socket" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:03.114492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.114466    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-podinfo" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:03.114836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.114809    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-kube-api-access-jz8lm" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "kube-api-access-jz8lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:03.114994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.114972    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-token" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:03.115691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.115054    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-envoy" pod "4028e239-916a-4474-9da9-bb2f47dfb84d" (UID: "4028e239-916a-4474-9da9-bb2f47dfb84d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:03.214799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.214770    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.214799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.214796    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.214799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.214805    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028e239-916a-4474-9da9-bb2f47dfb84d-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.215066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.214814    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028e239-916a-4474-9da9-bb2f47dfb84d-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.215066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.214822    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.215066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.214829    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz8lm\" (UniqueName: \"kubernetes.io/projected/4028e239-916a-4474-9da9-bb2f47dfb84d-kube-api-access-jz8lm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:03.527845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.527810    4576 generic.go:356] "Generic (PLEG): container finished" podID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerID="44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c" exitCode=0
Oct 30 01:17:03.528281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.527861    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"
Oct 30 01:17:03.528281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.527888    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" event={"ID":"4028e239-916a-4474-9da9-bb2f47dfb84d","Type":"ContainerDied","Data":"44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c"}
Oct 30 01:17:03.528281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.527940    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h" event={"ID":"4028e239-916a-4474-9da9-bb2f47dfb84d","Type":"ContainerDied","Data":"b39d0966e54ad7e69124ee8fe95a8e55fb364b1f8d025079038f468c324f776a"}
Oct 30 01:17:03.528281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.527965    4576 scope.go:122] "RemoveContainer" containerID="44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c"
Oct 30 01:17:03.539700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.539683    4576 scope.go:122] "RemoveContainer" containerID="e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c"
Oct 30 01:17:03.551278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.551252    4576 scope.go:122] "RemoveContainer" containerID="44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c"
Oct 30 01:17:03.551566 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:17:03.551546    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c\": container with ID starting with 44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c not found: ID does not exist" containerID="44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c"
Oct 30 01:17:03.551637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.551575    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c"} err="failed to get container status \"44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c\": rpc error: code = NotFound desc = could not find container \"44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c\": container with ID starting with 44ce30d2e157845ea6d007566ebfc9cabee16b83f3d6e3bdd479f45ca47f4d4c not found: ID does not exist"
Oct 30 01:17:03.551637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.551593    4576 scope.go:122] "RemoveContainer" containerID="e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c"
Oct 30 01:17:03.551827 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:17:03.551811    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c\": container with ID starting with e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c not found: ID does not exist" containerID="e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c"
Oct 30 01:17:03.551874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.551834    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c"} err="failed to get container status \"e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c\": rpc error: code = NotFound desc = could not find container \"e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c\": container with ID starting with e18353659f5baa643616ac575eece9956867f945acd9a6d89723c3ac5b19c56c not found: ID does not exist"
Oct 30 01:17:03.587884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.587857    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"]
Oct 30 01:17:03.600955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.600925    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b312f7-openshift-default-b5dc989c-ggh5h"]
Oct 30 01:17:03.885350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:03.885252    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:04.109625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:04.109587    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" path="/var/lib/kubelet/pods/4028e239-916a-4474-9da9-bb2f47dfb84d/volumes"
Oct 30 01:17:04.110078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:04.110063    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6576139-2679-4a30-b6dd-ec97d4dd6a41" path="/var/lib/kubelet/pods/e6576139-2679-4a30-b6dd-ec97d4dd6a41/volumes"
Oct 30 01:17:05.369466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:05.369431    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:06.385315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:06.385280    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"]
Oct 30 01:17:06.439042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:06.437558    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:06.934122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:06.934084    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:07.052370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:07.052338    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:07.442078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:07.442023    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:07.703822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:07.703711    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:08.770452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:08.770416    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:08.882807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:08.882771    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:08.998988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:08.998946    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:11.580486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:11.580446    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:11.713707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:11.713671    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:11.835062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:11.834966    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:13.190694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:13.190656    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:13.317143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:13.317107    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:13.532934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:13.532848    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:13.733535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:13.733501    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:13.888468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:13.888391    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:14.031329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:14.031294    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"]
Oct 30 01:17:14.046877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:14.046836    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8da3a029-lcb9m"]
Oct 30 01:17:14.110547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:14.110509    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e87c176-6d2e-47cd-a5c5-73075381d8f7" path="/var/lib/kubelet/pods/4e87c176-6d2e-47cd-a5c5-73075381d8f7/volumes"
Oct 30 01:17:14.221446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:14.221397    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:14.456315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:14.454196    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:14.676993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:14.676897    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps" podUID="d229ebd4-d9ec-4495-bb5e-476328755569" containerName="main" containerID="cri-o://5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a" gracePeriod=45
Oct 30 01:17:14.985697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:14.985667    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:17:15.089036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089000    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d229ebd4-d9ec-4495-bb5e-476328755569-tls-certs\" (UniqueName: \"kubernetes.io/secret/d229ebd4-d9ec-4495-bb5e-476328755569-tls-certs\") pod \"d229ebd4-d9ec-4495-bb5e-476328755569\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") "
Oct 30 01:17:15.089036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089040    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-home\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-home\") pod \"d229ebd4-d9ec-4495-bb5e-476328755569\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") "
Oct 30 01:17:15.089350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089094    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d229ebd4-d9ec-4495-bb5e-476328755569-kube-api-access-775v9\" (UniqueName: \"kubernetes.io/projected/d229ebd4-d9ec-4495-bb5e-476328755569-kube-api-access-775v9\") pod \"d229ebd4-d9ec-4495-bb5e-476328755569\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") "
Oct 30 01:17:15.089350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089160    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-dshm\") pod \"d229ebd4-d9ec-4495-bb5e-476328755569\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") "
Oct 30 01:17:15.089350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089182    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-model-cache\") pod \"d229ebd4-d9ec-4495-bb5e-476328755569\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") "
Oct 30 01:17:15.089350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089282    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-tmp-dir\") pod \"d229ebd4-d9ec-4495-bb5e-476328755569\" (UID: \"d229ebd4-d9ec-4495-bb5e-476328755569\") "
Oct 30 01:17:15.089596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089424    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-home" pod "d229ebd4-d9ec-4495-bb5e-476328755569" (UID: "d229ebd4-d9ec-4495-bb5e-476328755569"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:15.089596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089461    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-model-cache" pod "d229ebd4-d9ec-4495-bb5e-476328755569" (UID: "d229ebd4-d9ec-4495-bb5e-476328755569"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:15.089766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089740    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-tmp-dir" pod "d229ebd4-d9ec-4495-bb5e-476328755569" (UID: "d229ebd4-d9ec-4495-bb5e-476328755569"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:15.089910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089892    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:15.089979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089913    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:15.089979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.089945    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:15.091707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.091675    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d229ebd4-d9ec-4495-bb5e-476328755569-kube-api-access-775v9" pod "d229ebd4-d9ec-4495-bb5e-476328755569" (UID: "d229ebd4-d9ec-4495-bb5e-476328755569"). InnerVolumeSpecName "kube-api-access-775v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:15.091974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.091942    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d229ebd4-d9ec-4495-bb5e-476328755569-tls-certs" pod "d229ebd4-d9ec-4495-bb5e-476328755569" (UID: "d229ebd4-d9ec-4495-bb5e-476328755569"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:15.092038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.092026    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-dshm" pod "d229ebd4-d9ec-4495-bb5e-476328755569" (UID: "d229ebd4-d9ec-4495-bb5e-476328755569"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:15.192091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.192060    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d229ebd4-d9ec-4495-bb5e-476328755569-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:15.192091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.192090    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-775v9\" (UniqueName: \"kubernetes.io/projected/d229ebd4-d9ec-4495-bb5e-476328755569-kube-api-access-775v9\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:15.192351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.192103    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d229ebd4-d9ec-4495-bb5e-476328755569-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:15.726187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.726153    4576 generic.go:356] "Generic (PLEG): container finished" podID="d229ebd4-d9ec-4495-bb5e-476328755569" containerID="5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a" exitCode=0
Oct 30 01:17:15.726722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.726202    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"
Oct 30 01:17:15.726722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.726251    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps" event={"ID":"d229ebd4-d9ec-4495-bb5e-476328755569","Type":"ContainerDied","Data":"5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a"}
Oct 30 01:17:15.726722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.726296    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps" event={"ID":"d229ebd4-d9ec-4495-bb5e-476328755569","Type":"ContainerDied","Data":"ca8cb323f33629292f961422aa892d0cae1ce03cc8ccd959b48a5985d8b8b458"}
Oct 30 01:17:15.726722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.726315    4576 scope.go:122] "RemoveContainer" containerID="5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a"
Oct 30 01:17:15.738343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.738219    4576 scope.go:122] "RemoveContainer" containerID="5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a"
Oct 30 01:17:15.738681 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:17:15.738649    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a\": container with ID starting with 5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a not found: ID does not exist" containerID="5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a"
Oct 30 01:17:15.738816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.738703    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a"} err="failed to get container status \"5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a\": rpc error: code = NotFound desc = could not find container \"5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a\": container with ID starting with 5cd5f22b37fd592e340f9e0a2ae62cf5249c9ab1af40375c9e6c6b4dd867716a not found: ID does not exist"
Oct 30 01:17:15.784110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.784077    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"]
Oct 30 01:17:15.797288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:15.797257    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-b05110/auth-test-model-b05110-kserve-7bf77f56b5-clbps"]
Oct 30 01:17:16.010827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:16.010742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:16.111372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:16.111339    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d229ebd4-d9ec-4495-bb5e-476328755569" path="/var/lib/kubelet/pods/d229ebd4-d9ec-4495-bb5e-476328755569/volumes"
Oct 30 01:17:17.577505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:17.577470    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:18.382691 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:17:18.382656    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:17:18.563792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:18.563765    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:18.682664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:18.682582    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:19.266139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:19.266101    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:20.227178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.227143    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:20.341293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.341257    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:20.465796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.465762    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"]
Oct 30 01:17:20.466823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466789    4576 cpu_manager.go:414] "containerMap: removing container" podUID="d229ebd4-d9ec-4495-bb5e-476328755569" containerName="main"
Oct 30 01:17:20.466823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466819    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d229ebd4-d9ec-4495-bb5e-476328755569" containerName="main"
Oct 30 01:17:20.467044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466841    4576 cpu_manager.go:414] "containerMap: removing container" podUID="12b72cbc-9268-4b5f-bece-5a90b34dec79" containerName="revoke-keys"
Oct 30 01:17:20.467044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466851    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b72cbc-9268-4b5f-bece-5a90b34dec79" containerName="revoke-keys"
Oct 30 01:17:20.467044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466880    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e6576139-2679-4a30-b6dd-ec97d4dd6a41" containerName="payload-pre-processing"
Oct 30 01:17:20.467044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466892    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6576139-2679-4a30-b6dd-ec97d4dd6a41" containerName="payload-pre-processing"
Oct 30 01:17:20.467044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466944    4576 cpu_manager.go:414] "containerMap: removing container" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy"
Oct 30 01:17:20.467044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466954    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy"
Oct 30 01:17:20.467044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466976    4576 cpu_manager.go:414] "containerMap: removing container" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy"
Oct 30 01:17:20.467044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.466988    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy"
Oct 30 01:17:20.467527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.467168    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy"
Oct 30 01:17:20.467527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.467185    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="4028e239-916a-4474-9da9-bb2f47dfb84d" containerName="istio-proxy"
Oct 30 01:17:20.467527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.467202    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="d229ebd4-d9ec-4495-bb5e-476328755569" containerName="main"
Oct 30 01:17:20.467527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.467213    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b72cbc-9268-4b5f-bece-5a90b34dec79" containerName="revoke-keys"
Oct 30 01:17:20.467527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.467223    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6576139-2679-4a30-b6dd-ec97d4dd6a41" containerName="payload-pre-processing"
Oct 30 01:17:20.501075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.478502    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:20.531589 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.531285    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1075cb\""
Oct 30 01:17:20.531589 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.531437    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1075cb-dockercfg-rp45w\""
Oct 30 01:17:20.569607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.569073    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824c9aa8-ff1d-415c-90bb-7f0ac251a934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8\" (UID: \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:20.569607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.569215    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gff46\" (UniqueName: \"kubernetes.io/projected/824c9aa8-ff1d-415c-90bb-7f0ac251a934-kube-api-access-gff46\") pod \"payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8\" (UID: \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:20.618365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.618326    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"]
Oct 30 01:17:20.670971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.670939    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824c9aa8-ff1d-415c-90bb-7f0ac251a934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8\" (UID: \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:20.671146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.671046    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gff46\" (UniqueName: \"kubernetes.io/projected/824c9aa8-ff1d-415c-90bb-7f0ac251a934-kube-api-access-gff46\") pod \"payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8\" (UID: \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:20.671698 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.671678    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824c9aa8-ff1d-415c-90bb-7f0ac251a934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8\" (UID: \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:20.691552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.691527    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gff46\" (UniqueName: \"kubernetes.io/projected/824c9aa8-ff1d-415c-90bb-7f0ac251a934-kube-api-access-gff46\") pod \"payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8\" (UID: \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:20.902716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:20.902630    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:21.199454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.199418    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:21.212363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.212337    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"]
Oct 30 01:17:21.213197 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:17:21.213163    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824c9aa8_ff1d_415c_90bb_7f0ac251a934.slice/crio-9b8d9cfd20ad7c67849e64f440ea5457e52502d8a2ba29cbbcf369a46a0512b2 WatchSource:0}: Error finding container 9b8d9cfd20ad7c67849e64f440ea5457e52502d8a2ba29cbbcf369a46a0512b2: Status 404 returned error can't find the container with id 9b8d9cfd20ad7c67849e64f440ea5457e52502d8a2ba29cbbcf369a46a0512b2
Oct 30 01:17:21.413518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.413483    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8" podUID="c0d7d74d-6fd7-449c-b254-36f24efb49c5" containerName="main" containerID="cri-o://13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480" gracePeriod=45
Oct 30 01:17:21.670714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.670688    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:17:21.786090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.785996    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tmp-dir\") pod \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") "
Oct 30 01:17:21.786090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786058    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-model-cache\") pod \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") "
Oct 30 01:17:21.786364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786094    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-dshm\") pod \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") "
Oct 30 01:17:21.786364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786316    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tls-certs\") pod \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") "
Oct 30 01:17:21.786364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786349    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d7d74d-6fd7-449c-b254-36f24efb49c5-kube-api-access-f75q2\" (UniqueName: \"kubernetes.io/projected/c0d7d74d-6fd7-449c-b254-36f24efb49c5-kube-api-access-f75q2\") pod \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") "
Oct 30 01:17:21.786539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786346    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tmp-dir" pod "c0d7d74d-6fd7-449c-b254-36f24efb49c5" (UID: "c0d7d74d-6fd7-449c-b254-36f24efb49c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:21.786539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786392    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-home\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-home\") pod \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\" (UID: \"c0d7d74d-6fd7-449c-b254-36f24efb49c5\") "
Oct 30 01:17:21.786539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786421    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-model-cache" pod "c0d7d74d-6fd7-449c-b254-36f24efb49c5" (UID: "c0d7d74d-6fd7-449c-b254-36f24efb49c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:21.786694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786589    4576 generic.go:356] "Generic (PLEG): container finished" podID="c0d7d74d-6fd7-449c-b254-36f24efb49c5" containerID="13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480" exitCode=0
Oct 30 01:17:21.786694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786661    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8" event={"ID":"c0d7d74d-6fd7-449c-b254-36f24efb49c5","Type":"ContainerDied","Data":"13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480"}
Oct 30 01:17:21.786788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786698    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"
Oct 30 01:17:21.786788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786707    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-home" pod "c0d7d74d-6fd7-449c-b254-36f24efb49c5" (UID: "c0d7d74d-6fd7-449c-b254-36f24efb49c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:21.786788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786715    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8" event={"ID":"c0d7d74d-6fd7-449c-b254-36f24efb49c5","Type":"ContainerDied","Data":"47965450c5c5b0ab453a425e52dec8c165f5247bf8a3a808feb46e4f393d982b"}
Oct 30 01:17:21.786788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.786733    4576 scope.go:122] "RemoveContainer" containerID="13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480"
Oct 30 01:17:21.787167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.787132    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.787167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.787159    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.787329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.787172    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.789147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.789121    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8" event={"ID":"824c9aa8-ff1d-415c-90bb-7f0ac251a934","Type":"ContainerStarted","Data":"78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3"}
Oct 30 01:17:21.789324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.789176    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8" event={"ID":"824c9aa8-ff1d-415c-90bb-7f0ac251a934","Type":"ContainerStarted","Data":"9b8d9cfd20ad7c67849e64f440ea5457e52502d8a2ba29cbbcf369a46a0512b2"}
Oct 30 01:17:21.789324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.789204    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:21.789324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.789216    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d7d74d-6fd7-449c-b254-36f24efb49c5-kube-api-access-f75q2" pod "c0d7d74d-6fd7-449c-b254-36f24efb49c5" (UID: "c0d7d74d-6fd7-449c-b254-36f24efb49c5"). InnerVolumeSpecName "kube-api-access-f75q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:21.789324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.789316    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tls-certs" pod "c0d7d74d-6fd7-449c-b254-36f24efb49c5" (UID: "c0d7d74d-6fd7-449c-b254-36f24efb49c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:21.789530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.789341    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-dshm" pod "c0d7d74d-6fd7-449c-b254-36f24efb49c5" (UID: "c0d7d74d-6fd7-449c-b254-36f24efb49c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:21.818016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.817997    4576 scope.go:122] "RemoveContainer" containerID="13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480"
Oct 30 01:17:21.818408 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:17:21.818379    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480\": container with ID starting with 13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480 not found: ID does not exist" containerID="13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480"
Oct 30 01:17:21.818544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.818418    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480"} err="failed to get container status \"13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480\": rpc error: code = NotFound desc = could not find container \"13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480\": container with ID starting with 13258c72c6514630eb478823cccefb59990d60a987becaa737334dbed15c1480 not found: ID does not exist"
Oct 30 01:17:21.843706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.843649    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8" podStartSLOduration=1.8436328880000001 podStartE2EDuration="1.843632888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:21.84303867 +0000 UTC m=+2321.914362505" watchObservedRunningTime="2026-10-30 01:17:21.843632888 +0000 UTC m=+2321.914956723"
Oct 30 01:17:21.889033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.889002    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d7d74d-6fd7-449c-b254-36f24efb49c5-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.889033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.889037    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f75q2\" (UniqueName: \"kubernetes.io/projected/c0d7d74d-6fd7-449c-b254-36f24efb49c5-kube-api-access-f75q2\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:21.889223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:21.889053    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d7d74d-6fd7-449c-b254-36f24efb49c5-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:22.136300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:22.136267    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"]
Oct 30 01:17:22.139775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:22.139751    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4b973b/sub-test-model-4b973b-kserve-64df558b79-p6cl8"]
Oct 30 01:17:22.951431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:22.951395    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:24.110739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:24.110705    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d7d74d-6fd7-449c-b254-36f24efb49c5" path="/var/lib/kubelet/pods/c0d7d74d-6fd7-449c-b254-36f24efb49c5/volumes"
Oct 30 01:17:24.770214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:24.770179    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:26.028935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.028905    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"]
Oct 30 01:17:26.042085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.042051    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd8c650-m6k7c"]
Oct 30 01:17:26.109889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.109842    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f15448-71f1-4811-bd35-2c224f55d7eb" path="/var/lib/kubelet/pods/d7f15448-71f1-4811-bd35-2c224f55d7eb/volumes"
Oct 30 01:17:26.155322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.155293    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"]
Oct 30 01:17:26.155975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.155947    4576 cpu_manager.go:414] "containerMap: removing container" podUID="c0d7d74d-6fd7-449c-b254-36f24efb49c5" containerName="main"
Oct 30 01:17:26.155975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.155970    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d7d74d-6fd7-449c-b254-36f24efb49c5" containerName="main"
Oct 30 01:17:26.156186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.156097    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d7d74d-6fd7-449c-b254-36f24efb49c5" containerName="main"
Oct 30 01:17:26.161897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.161710    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.181160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.181129    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-43d152\"/\"openshift-service-ca.crt\""
Oct 30 01:17:26.185432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.185410    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-43d152\"/\"kube-root-ca.crt\""
Oct 30 01:17:26.187335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.187305    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-43d152\"/\"default-dockercfg-gk4jd\""
Oct 30 01:17:26.187448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.187407    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-43d152\"/\"sub-test-model-43d152-kserve-self-signed-certs\""
Oct 30 01:17:26.200808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.200773    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"]
Oct 30 01:17:26.236968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.236927    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:26.350486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.350396    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tmp-dir\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.350486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.350455    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tls-certs\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.350714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.350574    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-home\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.350793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.350767    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ctf\" (UniqueName: \"kubernetes.io/projected/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-kube-api-access-98ctf\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.350918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.350884    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-dshm\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.351051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.350919    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-model-cache\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.402719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.402688    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:26.451827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.451792    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-dshm\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.451827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.451829    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-model-cache\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.452110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.451905    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tmp-dir\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.452110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.451958    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tls-certs\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.452110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.451997    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-home\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.452268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.452132    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ctf\" (UniqueName: \"kubernetes.io/projected/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-kube-api-access-98ctf\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.453056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.452807    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-model-cache\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.453190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.453064    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-home\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.453264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.453218    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tmp-dir\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.458944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.458920    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tls-certs\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.459147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.459128    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-dshm\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.494019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.493977    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ctf\" (UniqueName: \"kubernetes.io/projected/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-kube-api-access-98ctf\") pod \"sub-test-model-43d152-kserve-595f57d84c-jnxfs\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") " pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.494487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.494462    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:26.848139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:26.848105    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"]
Oct 30 01:17:26.849158 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:17:26.849112    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d5ec9dc_591c_43da_a9fa_ed124cebc38e.slice/crio-ffb510d450f38191ab0f20a433accbd6b3d734476b4cfeea89df0d5930663348 WatchSource:0}: Error finding container ffb510d450f38191ab0f20a433accbd6b3d734476b4cfeea89df0d5930663348: Status 404 returned error can't find the container with id ffb510d450f38191ab0f20a433accbd6b3d734476b4cfeea89df0d5930663348
Oct 30 01:17:27.434172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.434140    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"]
Oct 30 01:17:27.441731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.439471    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.461159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.461128    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"]
Oct 30 01:17:27.470468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.470436    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-b05110-openshift-default-dockercfg-jlvbk\""
Oct 30 01:17:27.472224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472195    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jrn\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-kube-api-access-s5jrn\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.472419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472307    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istiod-ca-cert\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.472483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472450    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-token\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.472540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472508    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-podinfo\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.472601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472546    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-certs\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.472601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472579    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-data\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.472748 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472722    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.472821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472804    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-envoy\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.472881 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.472834    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-credential-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.574871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.574836    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-token\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.574871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.574872    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-podinfo\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.574920    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-certs\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.574963    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-data\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575007    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575057    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-envoy\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575091    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-credential-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575213    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jrn\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-kube-api-access-s5jrn\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575274    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istiod-ca-cert\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575390    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-certs\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575400    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-data\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575762    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-credential-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.575947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575930    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istiod-ca-cert\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.576007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.575948    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-socket\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.578161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.578138    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-envoy\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.578458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.578440    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-podinfo\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.595855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.595826    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-token\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.596100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.596084    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jrn\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-kube-api-access-s5jrn\") pod \"e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") " pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.763887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.763853    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:27.983259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.982682    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs" event={"ID":"3d5ec9dc-591c-43da-a9fa-ed124cebc38e","Type":"ContainerStarted","Data":"2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612"}
Oct 30 01:17:27.983259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:27.982742    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs" event={"ID":"3d5ec9dc-591c-43da-a9fa-ed124cebc38e","Type":"ContainerStarted","Data":"ffb510d450f38191ab0f20a433accbd6b3d734476b4cfeea89df0d5930663348"}
Oct 30 01:17:28.027694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:28.027603    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs" podStartSLOduration=2.027588166 podStartE2EDuration="2.027588166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:28.025817018 +0000 UTC m=+2328.097140859" watchObservedRunningTime="2026-10-30 01:17:28.027588166 +0000 UTC m=+2328.098912001"
Oct 30 01:17:28.081647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:28.081617    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"]
Oct 30 01:17:28.082410 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:17:28.082385    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ba917c_f190_46e5_9a6a_094f37d98c6a.slice/crio-73a69d5283d758d1383fbf01ff82096e2061a0c974aef5faf51a78ae176e0a2d WatchSource:0}: Error finding container 73a69d5283d758d1383fbf01ff82096e2061a0c974aef5faf51a78ae176e0a2d: Status 404 returned error can't find the container with id 73a69d5283d758d1383fbf01ff82096e2061a0c974aef5faf51a78ae176e0a2d
Oct 30 01:17:28.085167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:28.085134    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:17:28.085276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:28.085199    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:17:28.085276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:28.085227    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:17:28.186317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:28.186285    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:28.494753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:28.494721    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:28.998331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:28.998290    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" event={"ID":"a5ba917c-f190-46e5-9a6a-094f37d98c6a","Type":"ContainerStarted","Data":"57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1"}
Oct 30 01:17:29.000794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:29.000757    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" event={"ID":"a5ba917c-f190-46e5-9a6a-094f37d98c6a","Type":"ContainerStarted","Data":"73a69d5283d758d1383fbf01ff82096e2061a0c974aef5faf51a78ae176e0a2d"}
Oct 30 01:17:29.057215 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:29.057159    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" podStartSLOduration=2.057143043 podStartE2EDuration="2.057143043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:29.055014033 +0000 UTC m=+2329.126337883" watchObservedRunningTime="2026-10-30 01:17:29.057143043 +0000 UTC m=+2329.128466952"
Oct 30 01:17:29.585258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:29.582151    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:29.764969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:29.764932    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:29.766537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:29.766517    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:17:29.766599 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:29.766563    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 01:17:29.927486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:29.927408    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:30.764717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:30.764684    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:17:30.765199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:30.764742    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 01:17:31.297257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:31.290180    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:31.809341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:31.802619    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:17:31.809341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:31.802690    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 01:17:31.950803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:31.950763    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:32.044753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.044714    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"]
Oct 30 01:17:32.057856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.057825    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95fe33-5wf82"]
Oct 30 01:17:32.097310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.094799    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:32.108955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.108926    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b72cbc-9268-4b5f-bece-5a90b34dec79" path="/var/lib/kubelet/pods/12b72cbc-9268-4b5f-bece-5a90b34dec79/volumes"
Oct 30 01:17:32.202879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.202837    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:32.528201 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.528166    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:32.764598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.764567    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:17:32.764776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.764627    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 01:17:32.809995 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:32.809926    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:17:33.656627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:33.656586    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:33.764526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:33.764485    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:17:33.764714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:33.764550    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 01:17:34.764922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:34.764878    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 01:17:34.765336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:34.764948    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 01:17:35.247521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:35.247480    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:35.769740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:35.769701    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:35.770292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:35.770163    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:35.771155 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:35.771133    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:17:36.282814 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.282779    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"]
Oct 30 01:17:36.287048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.287028    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.303652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.303318    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-cfe8f0-openshift-default-dockercfg-4g4ql\""
Oct 30 01:17:36.321251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.321204    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"]
Oct 30 01:17:36.345199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.345162    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.345441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.345360    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-credential-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.345547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.345522    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c326c-dc0c-4720-b2f4-7e100846e52a-istiod-ca-cert\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.345620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.345576    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt9pk\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-kube-api-access-tt9pk\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.345692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.345623    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-token\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.345782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.345765    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-podinfo\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.345900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.345884    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-envoy\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.346004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.345988    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-certs\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.346072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.346060    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-data\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.447603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447572    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.447806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447641    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-credential-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.447806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447694    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c326c-dc0c-4720-b2f4-7e100846e52a-istiod-ca-cert\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.447806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447717    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt9pk\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-kube-api-access-tt9pk\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.447806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447736    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-token\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.447806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447772    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-podinfo\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.448121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447834    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-envoy\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.448121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447922    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-certs\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.448121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.447956    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-data\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.448121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.448086    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.448121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.448096    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-credential-socket\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.448438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.448352    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-data\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.448438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.448419    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-certs\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.448523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.448483    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c326c-dc0c-4720-b2f4-7e100846e52a-istiod-ca-cert\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.450356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.450333    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-envoy\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.450490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.450472    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-podinfo\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.468925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.468895    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-token\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.469100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.469078    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt9pk\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-kube-api-access-tt9pk\") pod \"e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") " pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.495128 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.495099    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:36.508646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.508616    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:36.602994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.602906    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:36.936198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.936164    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"]
Oct 30 01:17:36.936786 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:17:36.936761    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083c326c_dc0c_4720_b2f4_7e100846e52a.slice/crio-4dcc1a77e98036c3f6dbec3397f024b7e4f5cad03d0c10c6efdeb65d517eb2e2 WatchSource:0}: Error finding container 4dcc1a77e98036c3f6dbec3397f024b7e4f5cad03d0c10c6efdeb65d517eb2e2: Status 404 returned error can't find the container with id 4dcc1a77e98036c3f6dbec3397f024b7e4f5cad03d0c10c6efdeb65d517eb2e2
Oct 30 01:17:36.939999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.939957    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:17:36.940099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.940047    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:17:36.940099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.940093    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:17:36.995413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:36.995380    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:37.095371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:37.094130    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" event={"ID":"083c326c-dc0c-4720-b2f4-7e100846e52a","Type":"ContainerStarted","Data":"64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48"}
Oct 30 01:17:37.095371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:37.094187    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" event={"ID":"083c326c-dc0c-4720-b2f4-7e100846e52a","Type":"ContainerStarted","Data":"4dcc1a77e98036c3f6dbec3397f024b7e4f5cad03d0c10c6efdeb65d517eb2e2"}
Oct 30 01:17:37.114380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:37.114320    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:17:37.141337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:37.141281    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" podStartSLOduration=1.141266612 podStartE2EDuration="1.141266612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:37.138779405 +0000 UTC m=+2337.210103239" watchObservedRunningTime="2026-10-30 01:17:37.141266612 +0000 UTC m=+2337.212590437"
Oct 30 01:17:37.604391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:37.604357    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:38.606014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:38.605977    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 01:17:38.606571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:38.606048    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 01:17:38.635323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:38.635286    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:38.753116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:38.753078    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:39.011787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.011753    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"]
Oct 30 01:17:39.012788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.012756    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy" containerID="cri-o://7a222c104121cb157359eaff6fcce0606b289b97b14ebb355fa60159d9ecae5a" gracePeriod=30
Oct 30 01:17:39.042472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.042424    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:39.295803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.295699    4576 patch_prober.go:29] interesting pod/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:17:39.295803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.295759    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:17:39.603798 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.603706    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 01:17:39.603971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.603822    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 01:17:39.695257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.694626    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:39.801789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:39.801750    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:40.245176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:40.245046    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:40.357108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:40.357062    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:40.480125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:40.480094    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:40.603709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:40.603611    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 01:17:40.603978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:40.603947    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 01:17:41.175261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:41.173736    4576 scope.go:122] "RemoveContainer" containerID="fde9ae916b7d4006be30bffadc9fdcc72307dfec29cca88a23ac981b73923cb4"
Oct 30 01:17:41.197602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:41.197572    4576 scope.go:122] "RemoveContainer" containerID="2b70252732fe4b61aeb0c83cfd9fca50fd9bac6ca637df20bdaf6a463544a839"
Oct 30 01:17:41.217474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:41.217452    4576 scope.go:122] "RemoveContainer" containerID="17ce80a3ad2b192867c4e409717bdea94a739951940288cc5edf42b35fcb3674"
Oct 30 01:17:41.227902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:41.227878    4576 scope.go:122] "RemoveContainer" containerID="6418d1eacdcd8bc63a2b40e9a1a165c0abe7e6e35ac0449b494acd0be66ed00e"
Oct 30 01:17:41.508274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:41.508192    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:41.606444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:41.605792    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 01:17:41.606444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:41.605848    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 01:17:42.196257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:42.193762    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:42.318924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:42.318887    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:42.634124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:42.634003    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 01:17:42.634459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:42.634346    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 01:17:43.681085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:43.681037    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:43.681966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:43.681944    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:43.682793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:43.682769    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:17:44.115645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.115612    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:44.213460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.213413    4576 generic.go:356] "Generic (PLEG): container finished" podID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerID="7a222c104121cb157359eaff6fcce0606b289b97b14ebb355fa60159d9ecae5a" exitCode=0
Oct 30 01:17:44.213693 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.213484    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" event={"ID":"90e587ab-e7ae-4579-a632-ea1a760636bb","Type":"ContainerDied","Data":"7a222c104121cb157359eaff6fcce0606b289b97b14ebb355fa60159d9ecae5a"}
Oct 30 01:17:44.262614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.262579    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:44.311979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.311954    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:17:44.414260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.411848    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:44.476264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.473099    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-certs" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:44.476264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.475030    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-certs\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.476264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.475119    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-token\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-token\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.476264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.475678    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e587ab-e7ae-4579-a632-ea1a760636bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e587ab-e7ae-4579-a632-ea1a760636bb-istiod-ca-cert\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.476264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.476026    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e587ab-e7ae-4579-a632-ea1a760636bb-istiod-ca-cert" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:44.476264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.476100    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-data\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.476754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.476380    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-data" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:44.476754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.476484    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-podinfo\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.476907    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-envoy\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.477441    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-credential-socket\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.477472    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-socket\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.477699    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-credential-socket" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.477834    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-socket" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.477959    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-kube-api-access-fdgp2\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-kube-api-access-fdgp2\") pod \"90e587ab-e7ae-4579-a632-ea1a760636bb\" (UID: \"90e587ab-e7ae-4579-a632-ea1a760636bb\") "
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.478846    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.478867    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.478882    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.478893    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e587ab-e7ae-4579-a632-ea1a760636bb-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:44.479973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.478904    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:44.496026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.492613    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-kube-api-access-fdgp2" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "kube-api-access-fdgp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:44.496026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.492780    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-envoy" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:44.496026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.492905    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-podinfo" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:44.499428 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.499365    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-token" pod "90e587ab-e7ae-4579-a632-ea1a760636bb" (UID: "90e587ab-e7ae-4579-a632-ea1a760636bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:44.580326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.580196    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:44.580326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.580226    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:44.580326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.580254    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e587ab-e7ae-4579-a632-ea1a760636bb-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:44.580326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:44.580265    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdgp2\" (UniqueName: \"kubernetes.io/projected/90e587ab-e7ae-4579-a632-ea1a760636bb-kube-api-access-fdgp2\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:17:45.228721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:45.228679    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp" event={"ID":"90e587ab-e7ae-4579-a632-ea1a760636bb","Type":"ContainerDied","Data":"bc87646c2263d3d6f543c9d9646e260257bf6a8fae18fc9b8e6abb6a0605c3c8"}
Oct 30 01:17:45.229317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:45.228738    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"
Oct 30 01:17:45.229317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:45.228768    4576 scope.go:122] "RemoveContainer" containerID="7a222c104121cb157359eaff6fcce0606b289b97b14ebb355fa60159d9ecae5a"
Oct 30 01:17:45.302409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:45.302372    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"]
Oct 30 01:17:45.305713 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:45.305687    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-95fe33-openshift-default-66865bdbbc-d9smp"]
Oct 30 01:17:45.940402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:45.940355    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:46.066042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:46.066010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:46.116408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:46.116367    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" path="/var/lib/kubelet/pods/90e587ab-e7ae-4579-a632-ea1a760636bb/volumes"
Oct 30 01:17:46.303340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:46.301612    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:47.919412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:47.919374    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:48.022491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:48.022461    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:48.135567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:48.135529    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:48.382827 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:17:48.382788    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:17:49.636301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:49.636267    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:51.107344 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:51.107304    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:52.563996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:52.563962    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:54.133042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:54.133006    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:54.244166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:54.244126    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:54.351467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:54.351433    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:55.725958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:55.725925    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:55.728009 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:55.727987    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:17:57.424206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:57.424169    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:58.889178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:58.889148    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"]
Oct 30 01:17:58.889773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:58.889758    4576 cpu_manager.go:414] "containerMap: removing container" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy"
Oct 30 01:17:58.889815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:58.889774    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy"
Oct 30 01:17:58.889901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:58.889892    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e587ab-e7ae-4579-a632-ea1a760636bb" containerName="istio-proxy"
Oct 30 01:17:58.895120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:58.895098    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:58.919479 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:58.919448    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"]
Oct 30 01:17:58.919769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:58.919492    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:58.922433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:58.922413    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:17:59.004793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.004757    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.005077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.005050    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7d4\" (UniqueName: \"kubernetes.io/projected/2e63a1cd-fe40-4777-97a8-17f7bd021f74-kube-api-access-vr7d4\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.005158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.005093    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.106195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.106162    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7d4\" (UniqueName: \"kubernetes.io/projected/2e63a1cd-fe40-4777-97a8-17f7bd021f74-kube-api-access-vr7d4\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.106442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.106201    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.106442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.106316    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.106872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.106847    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.108901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.108878    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.128709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.128683    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7d4\" (UniqueName: \"kubernetes.io/projected/2e63a1cd-fe40-4777-97a8-17f7bd021f74-kube-api-access-vr7d4\") pod \"maas-api-revoke-sub-eaf7777d-2mf2j\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.202383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.202344    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:17:59.219048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.219025    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:17:59.546924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:17:59.546884    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"]
Oct 30 01:18:00.426303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:00.426268    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j" event={"ID":"2e63a1cd-fe40-4777-97a8-17f7bd021f74","Type":"ContainerStarted","Data":"b230ef6d9e5d42d6b54178739780d5a8e8fb2709399ea80b3368e76bbc229fe8"}
Oct 30 01:18:00.426831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:00.426326    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j" event={"ID":"2e63a1cd-fe40-4777-97a8-17f7bd021f74","Type":"ContainerStarted","Data":"1c94dc320310a09fdcd9db5811daa3fc9b4c99e0619a3371fa5d10367369b572"}
Oct 30 01:18:00.468811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:00.468762    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j" podStartSLOduration=2.46874705 podStartE2EDuration="2.46874705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:00.46640383 +0000 UTC m=+2360.537727664" watchObservedRunningTime="2026-10-30 01:18:00.46874705 +0000 UTC m=+2360.540070884"
Oct 30 01:18:00.734796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:00.734761    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:01.436547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:01.436513    4576 generic.go:356] "Generic (PLEG): container finished" podID="2e63a1cd-fe40-4777-97a8-17f7bd021f74" containerID="b230ef6d9e5d42d6b54178739780d5a8e8fb2709399ea80b3368e76bbc229fe8" exitCode=0
Oct 30 01:18:01.436983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:01.436603    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j" event={"ID":"2e63a1cd-fe40-4777-97a8-17f7bd021f74","Type":"ContainerDied","Data":"b230ef6d9e5d42d6b54178739780d5a8e8fb2709399ea80b3368e76bbc229fe8"}
Oct 30 01:18:02.359947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.359906    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:02.613777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.613720    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:18:02.763050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.763020    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-xks-ca\") pod \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") "
Oct 30 01:18:02.763415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.763392    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-openshift-ca\") pod \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") "
Oct 30 01:18:02.763680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.763658    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e63a1cd-fe40-4777-97a8-17f7bd021f74-kube-api-access-vr7d4\" (UniqueName: \"kubernetes.io/projected/2e63a1cd-fe40-4777-97a8-17f7bd021f74-kube-api-access-vr7d4\") pod \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\" (UID: \"2e63a1cd-fe40-4777-97a8-17f7bd021f74\") "
Oct 30 01:18:02.764648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.764619    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-openshift-ca" pod "2e63a1cd-fe40-4777-97a8-17f7bd021f74" (UID: "2e63a1cd-fe40-4777-97a8-17f7bd021f74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:02.766552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.766523    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-xks-ca" pod "2e63a1cd-fe40-4777-97a8-17f7bd021f74" (UID: "2e63a1cd-fe40-4777-97a8-17f7bd021f74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:02.766668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.766619    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e63a1cd-fe40-4777-97a8-17f7bd021f74-kube-api-access-vr7d4" pod "2e63a1cd-fe40-4777-97a8-17f7bd021f74" (UID: "2e63a1cd-fe40-4777-97a8-17f7bd021f74"). InnerVolumeSpecName "kube-api-access-vr7d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:02.865684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.865600    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:02.865684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.865635    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr7d4\" (UniqueName: \"kubernetes.io/projected/2e63a1cd-fe40-4777-97a8-17f7bd021f74-kube-api-access-vr7d4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:02.865684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:02.865650    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e63a1cd-fe40-4777-97a8-17f7bd021f74-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:03.269026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.268990    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"]
Oct 30 01:18:03.269927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.269902    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2e63a1cd-fe40-4777-97a8-17f7bd021f74" containerName="revoke-keys"
Oct 30 01:18:03.269927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.269928    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e63a1cd-fe40-4777-97a8-17f7bd021f74" containerName="revoke-keys"
Oct 30 01:18:03.270654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.270634    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e63a1cd-fe40-4777-97a8-17f7bd021f74" containerName="revoke-keys"
Oct 30 01:18:03.275116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.275094    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.297104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.297071    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"]
Oct 30 01:18:03.302894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.300320    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-ad00a0\"/\"openshift-service-ca.crt\""
Oct 30 01:18:03.302894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.300355    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-ad00a0\"/\"auth-test-model-ad00a0-kserve-self-signed-certs\""
Oct 30 01:18:03.302894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.300355    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-ad00a0\"/\"kube-root-ca.crt\""
Oct 30 01:18:03.312558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.312531    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-ad00a0\"/\"default-dockercfg-wq9zq\""
Oct 30 01:18:03.346865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.346830    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:03.373055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.373017    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tmp-dir\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.373316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.373114    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tls-certs\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.373316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.373168    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-model-cache\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.373316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.373222    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-home\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.373525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.373375    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-dshm\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.373525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.373487    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjf5\" (UniqueName: \"kubernetes.io/projected/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-kube-api-access-9hjf5\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.455838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.455810    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"
Oct 30 01:18:03.456006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.455842    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j" event={"ID":"2e63a1cd-fe40-4777-97a8-17f7bd021f74","Type":"ContainerDied","Data":"1c94dc320310a09fdcd9db5811daa3fc9b4c99e0619a3371fa5d10367369b572"}
Oct 30 01:18:03.456006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.455890    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c94dc320310a09fdcd9db5811daa3fc9b4c99e0619a3371fa5d10367369b572"
Oct 30 01:18:03.475249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.475206    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjf5\" (UniqueName: \"kubernetes.io/projected/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-kube-api-access-9hjf5\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.475424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.475402    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tmp-dir\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.475518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.475442    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tls-certs\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.475598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.475577    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-model-cache\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.475657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.475638    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-home\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.475722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.475710    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-dshm\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.475848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.475827    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tmp-dir\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.476119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.476082    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-model-cache\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.476388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.476369    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-home\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.478347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.478323    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tls-certs\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.478488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.478417    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-dshm\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.521311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.513932    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjf5\" (UniqueName: \"kubernetes.io/projected/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-kube-api-access-9hjf5\") pod \"auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") " pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.605831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.605792    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:03.922131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:03.922049    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"]
Oct 30 01:18:03.925966 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:18:03.925936    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddbd16dd_6a19_42e2_9406_fc4d963a53c2.slice/crio-4240cfbe0a86d3747e97cff679fe5fb67523ce8954f4fbf0f5b72caffc8c272d WatchSource:0}: Error finding container 4240cfbe0a86d3747e97cff679fe5fb67523ce8954f4fbf0f5b72caffc8c272d: Status 404 returned error can't find the container with id 4240cfbe0a86d3747e97cff679fe5fb67523ce8954f4fbf0f5b72caffc8c272d
Oct 30 01:18:04.028965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:04.028919    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"]
Oct 30 01:18:04.032903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:04.032869    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f37044b-jbf8m"]
Oct 30 01:18:04.057789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:04.057754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:04.117813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:04.117769    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cfa32f-a497-44dc-8591-22ae8c972694" path="/var/lib/kubelet/pods/26cfa32f-a497-44dc-8591-22ae8c972694/volumes"
Oct 30 01:18:04.464728 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:04.464687    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" event={"ID":"ddbd16dd-6a19-42e2-9406-fc4d963a53c2","Type":"ContainerStarted","Data":"696ccb1d5cd705629e6473626eabcf6d7f28886f84c579240349e10a06d362b4"}
Oct 30 01:18:04.464728 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:04.464733    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" event={"ID":"ddbd16dd-6a19-42e2-9406-fc4d963a53c2","Type":"ContainerStarted","Data":"4240cfbe0a86d3747e97cff679fe5fb67523ce8954f4fbf0f5b72caffc8c272d"}
Oct 30 01:18:04.535832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:04.535779    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" podStartSLOduration=1.535764317 podStartE2EDuration="1.535764317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:04.534316698 +0000 UTC m=+2364.605640546" watchObservedRunningTime="2026-10-30 01:18:04.535764317 +0000 UTC m=+2364.607088151"
Oct 30 01:18:04.606101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:04.606069    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:05.582463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:05.582426    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:07.236340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:07.236312    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:08.790670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:08.790627    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:09.027825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.027793    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"]
Oct 30 01:18:09.041485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.041403    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cd6e6d0-jhfzx"]
Oct 30 01:18:09.171994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.171957    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"]
Oct 30 01:18:09.176262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.176210    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.192667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.192642    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:18:09.192793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.192642    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:18:09.193859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.193838    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"]
Oct 30 01:18:09.247666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.247625    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckbpw\" (UniqueName: \"kubernetes.io/projected/b0bdfb97-a06f-449e-b603-e61f7cbecefd-kube-api-access-ckbpw\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.247881 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.247861    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.247949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.247907    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.348815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.348720    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.348815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.348781    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.349060 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.348878    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckbpw\" (UniqueName: \"kubernetes.io/projected/b0bdfb97-a06f-449e-b603-e61f7cbecefd-kube-api-access-ckbpw\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.349531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.349510    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.351370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.351351    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.370115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.370087    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckbpw\" (UniqueName: \"kubernetes.io/projected/b0bdfb97-a06f-449e-b603-e61f7cbecefd-kube-api-access-ckbpw\") pod \"maas-api-revoke-sub-a7c35562-dtz9g\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.490749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.490710    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:09.801928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:09.801896    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"]
Oct 30 01:18:09.802775 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:18:09.802748    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bdfb97_a06f_449e_b603_e61f7cbecefd.slice/crio-8f37db86f44f4242a10acb0bd1294f933be507d4177b4f6a2987584f41bdee6d WatchSource:0}: Error finding container 8f37db86f44f4242a10acb0bd1294f933be507d4177b4f6a2987584f41bdee6d: Status 404 returned error can't find the container with id 8f37db86f44f4242a10acb0bd1294f933be507d4177b4f6a2987584f41bdee6d
Oct 30 01:18:10.111533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:10.111489    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed" path="/var/lib/kubelet/pods/2aaae8a5-6ca0-484f-a6c7-b230eb34a7ed/volumes"
Oct 30 01:18:10.509719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:10.509683    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g" event={"ID":"b0bdfb97-a06f-449e-b603-e61f7cbecefd","Type":"ContainerStarted","Data":"60be0a938d0b0b43ec9a3a5914c0139b2c30c850cf3f92166768e1d6711ce997"}
Oct 30 01:18:10.509941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:10.509746    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g" event={"ID":"b0bdfb97-a06f-449e-b603-e61f7cbecefd","Type":"ContainerStarted","Data":"8f37db86f44f4242a10acb0bd1294f933be507d4177b4f6a2987584f41bdee6d"}
Oct 30 01:18:10.553616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:10.553567    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g" podStartSLOduration=1.5535521700000001 podStartE2EDuration="1.55355217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:10.552371048 +0000 UTC m=+2370.623694920" watchObservedRunningTime="2026-10-30 01:18:10.55355217 +0000 UTC m=+2370.624876004"
Oct 30 01:18:10.554676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:10.554642    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:11.518594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:11.518557    4576 generic.go:356] "Generic (PLEG): container finished" podID="b0bdfb97-a06f-449e-b603-e61f7cbecefd" containerID="60be0a938d0b0b43ec9a3a5914c0139b2c30c850cf3f92166768e1d6711ce997" exitCode=0
Oct 30 01:18:11.519002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:11.518640    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g" event={"ID":"b0bdfb97-a06f-449e-b603-e61f7cbecefd","Type":"ContainerDied","Data":"60be0a938d0b0b43ec9a3a5914c0139b2c30c850cf3f92166768e1d6711ce997"}
Oct 30 01:18:12.077560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.077517    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:12.682262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.682221    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:12.798189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.798099    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-xks-ca\") pod \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") "
Oct 30 01:18:12.798392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.798275    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-openshift-ca\") pod \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") "
Oct 30 01:18:12.798392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.798324    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bdfb97-a06f-449e-b603-e61f7cbecefd-kube-api-access-ckbpw\" (UniqueName: \"kubernetes.io/projected/b0bdfb97-a06f-449e-b603-e61f7cbecefd-kube-api-access-ckbpw\") pod \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\" (UID: \"b0bdfb97-a06f-449e-b603-e61f7cbecefd\") "
Oct 30 01:18:12.798649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.798626    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-openshift-ca" pod "b0bdfb97-a06f-449e-b603-e61f7cbecefd" (UID: "b0bdfb97-a06f-449e-b603-e61f7cbecefd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:12.798928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.798912    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:12.800555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.800532    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bdfb97-a06f-449e-b603-e61f7cbecefd-kube-api-access-ckbpw" pod "b0bdfb97-a06f-449e-b603-e61f7cbecefd" (UID: "b0bdfb97-a06f-449e-b603-e61f7cbecefd"). InnerVolumeSpecName "kube-api-access-ckbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:12.800622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.800572    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-xks-ca" pod "b0bdfb97-a06f-449e-b603-e61f7cbecefd" (UID: "b0bdfb97-a06f-449e-b603-e61f7cbecefd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:12.900370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.900304    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bdfb97-a06f-449e-b603-e61f7cbecefd-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:12.900370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:12.900344    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckbpw\" (UniqueName: \"kubernetes.io/projected/b0bdfb97-a06f-449e-b603-e61f7cbecefd-kube-api-access-ckbpw\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:13.534548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:13.534515    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g" event={"ID":"b0bdfb97-a06f-449e-b603-e61f7cbecefd","Type":"ContainerDied","Data":"8f37db86f44f4242a10acb0bd1294f933be507d4177b4f6a2987584f41bdee6d"}
Oct 30 01:18:13.534740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:13.534555    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"
Oct 30 01:18:13.534740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:13.534563    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f37db86f44f4242a10acb0bd1294f933be507d4177b4f6a2987584f41bdee6d"
Oct 30 01:18:13.606285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:13.606203    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:13.619743 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:13.619717    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:13.671298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:13.671264    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:14.553628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:14.553599    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:18:15.359022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:15.358962    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:16.796031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:16.795989    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:18.208098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:18.208059    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:18.383383 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:18:18.383345    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:18:19.028524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:19.028486    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"]
Oct 30 01:18:19.043228 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:19.043199    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e12a61a-sps94"]
Oct 30 01:18:19.810882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:19.810851    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:19.923148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:19.923115    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:20.031372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:20.031334    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:20.110117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:20.110042    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9" path="/var/lib/kubelet/pods/e1c8ee45-2b1e-4f16-ac5c-ddf0e8cfe2f9/volumes"
Oct 30 01:18:21.175453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:21.175406    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"]
Oct 30 01:18:21.217461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:21.217416    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:21.517385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:21.517350    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:22.987570 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:22.987528    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:24.541905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:24.541865    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:26.023124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:26.023078    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:27.118749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:27.118701    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:27.242503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:27.242464    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:27.652557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:27.652515    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:27.966281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:27.966227    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:28.090504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:28.090470    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:28.210593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:28.210544    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:29.313617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:29.313580    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:31.064860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:31.064829    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:32.665171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:32.665132    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:33.035003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:33.034966    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"]
Oct 30 01:18:33.047104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:33.047074    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d637624-shg29"]
Oct 30 01:18:34.110625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.110575    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ca165b-9d10-4098-b61f-698e2669f0ac" path="/var/lib/kubelet/pods/f2ca165b-9d10-4098-b61f-698e2669f0ac/volumes"
Oct 30 01:18:34.556977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.556945    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8889-73b17f"]
Oct 30 01:18:34.557590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.557573    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b0bdfb97-a06f-449e-b603-e61f7cbecefd" containerName="revoke-keys"
Oct 30 01:18:34.557652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.557592    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bdfb97-a06f-449e-b603-e61f7cbecefd" containerName="revoke-keys"
Oct 30 01:18:34.557710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.557701    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bdfb97-a06f-449e-b603-e61f7cbecefd" containerName="revoke-keys"
Oct 30 01:18:34.562543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.562522    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:34.576965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.576292    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8889-73b17f"]
Oct 30 01:18:34.585634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.585604    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-98k27\""
Oct 30 01:18:34.629628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.629582    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv5j\" (UniqueName: \"kubernetes.io/projected/75d995d3-375f-4320-bd40-dd8ca586bc8f-kube-api-access-bnv5j\") pod \"test-curl-8889-73b17f\" (UID: \"75d995d3-375f-4320-bd40-dd8ca586bc8f\") " pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:34.702988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.702951    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:34.731058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.731011    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv5j\" (UniqueName: \"kubernetes.io/projected/75d995d3-375f-4320-bd40-dd8ca586bc8f-kube-api-access-bnv5j\") pod \"test-curl-8889-73b17f\" (UID: \"75d995d3-375f-4320-bd40-dd8ca586bc8f\") " pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:34.752932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.752899    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv5j\" (UniqueName: \"kubernetes.io/projected/75d995d3-375f-4320-bd40-dd8ca586bc8f-kube-api-access-bnv5j\") pod \"test-curl-8889-73b17f\" (UID: \"75d995d3-375f-4320-bd40-dd8ca586bc8f\") " pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:34.892344 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:34.892221    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:35.166306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:35.166199    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"]
Oct 30 01:18:35.197034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:35.197004    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8889-73b17f"]
Oct 30 01:18:35.198219 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:18:35.198186    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75d995d3_375f_4320_bd40_dd8ca586bc8f.slice/crio-1ec9a69b502ee94a6ae8c1ae94935d5d105b91dadd1de0077b93be2eb6a4ac44 WatchSource:0}: Error finding container 1ec9a69b502ee94a6ae8c1ae94935d5d105b91dadd1de0077b93be2eb6a4ac44: Status 404 returned error can't find the container with id 1ec9a69b502ee94a6ae8c1ae94935d5d105b91dadd1de0077b93be2eb6a4ac44
Oct 30 01:18:35.200320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:35.200301    4576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:18:35.349069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:35.349040    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:35.467445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:35.467405    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:35.711826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:35.711794    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8889-73b17f" event={"ID":"75d995d3-375f-4320-bd40-dd8ca586bc8f","Type":"ContainerStarted","Data":"1ec9a69b502ee94a6ae8c1ae94935d5d105b91dadd1de0077b93be2eb6a4ac44"}
Oct 30 01:18:36.200812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.200772    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw" podUID="4d265226-b376-4d80-98b6-0f56ce3055cc" containerName="main" containerID="cri-o://0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c" gracePeriod=45
Oct 30 01:18:36.480844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.480818    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:18:36.562248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562198    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-tmp-dir\") pod \"4d265226-b376-4d80-98b6-0f56ce3055cc\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") "
Oct 30 01:18:36.562552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562292    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-model-cache\") pod \"4d265226-b376-4d80-98b6-0f56ce3055cc\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") "
Oct 30 01:18:36.562552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562409    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-dshm\") pod \"4d265226-b376-4d80-98b6-0f56ce3055cc\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") "
Oct 30 01:18:36.562552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562493    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-home\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-home\") pod \"4d265226-b376-4d80-98b6-0f56ce3055cc\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") "
Oct 30 01:18:36.562756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562608    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-model-cache" pod "4d265226-b376-4d80-98b6-0f56ce3055cc" (UID: "4d265226-b376-4d80-98b6-0f56ce3055cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:36.562756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562725    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-home" pod "4d265226-b376-4d80-98b6-0f56ce3055cc" (UID: "4d265226-b376-4d80-98b6-0f56ce3055cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:36.562855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562766    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d265226-b376-4d80-98b6-0f56ce3055cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d265226-b376-4d80-98b6-0f56ce3055cc-tls-certs\") pod \"4d265226-b376-4d80-98b6-0f56ce3055cc\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") "
Oct 30 01:18:36.562961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562933    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d265226-b376-4d80-98b6-0f56ce3055cc-kube-api-access-vn6n4\" (UniqueName: \"kubernetes.io/projected/4d265226-b376-4d80-98b6-0f56ce3055cc-kube-api-access-vn6n4\") pod \"4d265226-b376-4d80-98b6-0f56ce3055cc\" (UID: \"4d265226-b376-4d80-98b6-0f56ce3055cc\") "
Oct 30 01:18:36.563030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.562535    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-tmp-dir" pod "4d265226-b376-4d80-98b6-0f56ce3055cc" (UID: "4d265226-b376-4d80-98b6-0f56ce3055cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:36.563744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.563718    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:36.563890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.563747    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:36.563890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.563762    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:36.565342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.565316    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-dshm" pod "4d265226-b376-4d80-98b6-0f56ce3055cc" (UID: "4d265226-b376-4d80-98b6-0f56ce3055cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:36.565745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.565440    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d265226-b376-4d80-98b6-0f56ce3055cc-kube-api-access-vn6n4" pod "4d265226-b376-4d80-98b6-0f56ce3055cc" (UID: "4d265226-b376-4d80-98b6-0f56ce3055cc"). InnerVolumeSpecName "kube-api-access-vn6n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:36.565745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.565656    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d265226-b376-4d80-98b6-0f56ce3055cc-tls-certs" pod "4d265226-b376-4d80-98b6-0f56ce3055cc" (UID: "4d265226-b376-4d80-98b6-0f56ce3055cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:36.664851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.664820    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d265226-b376-4d80-98b6-0f56ce3055cc-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:36.664851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.664851    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d265226-b376-4d80-98b6-0f56ce3055cc-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:36.665069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.664865    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn6n4\" (UniqueName: \"kubernetes.io/projected/4d265226-b376-4d80-98b6-0f56ce3055cc-kube-api-access-vn6n4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:36.665929 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.665898    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:36.733184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.733089    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8889-73b17f" event={"ID":"75d995d3-375f-4320-bd40-dd8ca586bc8f","Type":"ContainerStarted","Data":"afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f"}
Oct 30 01:18:36.734985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.734953    4576 generic.go:356] "Generic (PLEG): container finished" podID="4d265226-b376-4d80-98b6-0f56ce3055cc" containerID="0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c" exitCode=0
Oct 30 01:18:36.735227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.735208    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"
Oct 30 01:18:36.735453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.735207    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw" event={"ID":"4d265226-b376-4d80-98b6-0f56ce3055cc","Type":"ContainerDied","Data":"0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c"}
Oct 30 01:18:36.735661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.735473    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw" event={"ID":"4d265226-b376-4d80-98b6-0f56ce3055cc","Type":"ContainerDied","Data":"7469c0bd612433057ae42e1a783186a3c6a9511cfb8bf9fd2d533168b6d7bc56"}
Oct 30 01:18:36.735661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.735509    4576 scope.go:122] "RemoveContainer" containerID="0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c"
Oct 30 01:18:36.752707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.752429    4576 scope.go:122] "RemoveContainer" containerID="0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c"
Oct 30 01:18:36.753626 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:18:36.753394    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c\": container with ID starting with 0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c not found: ID does not exist" containerID="0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c"
Oct 30 01:18:36.753626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.753440    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c"} err="failed to get container status \"0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c\": rpc error: code = NotFound desc = could not find container \"0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c\": container with ID starting with 0ae98b9ed7204c737f070f1fffd31db6653ff7109c00ce40041d7365d52bb76c not found: ID does not exist"
Oct 30 01:18:36.775359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.775296    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8889-73b17f" podStartSLOduration=2.110991935 podStartE2EDuration="2.775282508s" totalImagesPullingTime="664.290573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:36.772271984 +0000 UTC m=+2396.843595821" watchObservedRunningTime="2026-10-30 01:18:36.775282508 +0000 UTC m=+2396.846606343"
Oct 30 01:18:36.815267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.815218    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"]
Oct 30 01:18:36.820636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:36.820604    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-42d56b/rate-test-model-42d56b-kserve-6489dc4585-x6fvw"]
Oct 30 01:18:37.588822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.588789    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8889-73b17f"]
Oct 30 01:18:37.592811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.592777    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8889-73b17f"]
Oct 30 01:18:37.792267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.792202    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"]
Oct 30 01:18:37.792916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.792898    4576 cpu_manager.go:414] "containerMap: removing container" podUID="4d265226-b376-4d80-98b6-0f56ce3055cc" containerName="main"
Oct 30 01:18:37.792970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.792919    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d265226-b376-4d80-98b6-0f56ce3055cc" containerName="main"
Oct 30 01:18:37.792970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.792933    4576 cpu_manager.go:414] "containerMap: removing container" podUID="75d995d3-375f-4320-bd40-dd8ca586bc8f" containerName="test-curl-8889-73b17f"
Oct 30 01:18:37.792970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.792938    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d995d3-375f-4320-bd40-dd8ca586bc8f" containerName="test-curl-8889-73b17f"
Oct 30 01:18:37.793075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.793050    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d265226-b376-4d80-98b6-0f56ce3055cc" containerName="main"
Oct 30 01:18:37.793075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.793061    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d995d3-375f-4320-bd40-dd8ca586bc8f" containerName="test-curl-8889-73b17f"
Oct 30 01:18:37.798019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.798001    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:37.812642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.812616    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"]
Oct 30 01:18:37.820290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.814276    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:18:37.820290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.814387    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:18:37.881097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.881022    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpsl\" (UniqueName: \"kubernetes.io/projected/7bc377c2-639b-4d6a-b41f-bd51682b6913-kube-api-access-jkpsl\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:37.881285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.881193    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:37.881285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.881228    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:37.982645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.982607    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpsl\" (UniqueName: \"kubernetes.io/projected/7bc377c2-639b-4d6a-b41f-bd51682b6913-kube-api-access-jkpsl\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:37.982828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.982765    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:37.982828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.982800    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:37.983694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.983595    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:37.985454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:37.985429    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:38.003995 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:38.003972    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpsl\" (UniqueName: \"kubernetes.io/projected/7bc377c2-639b-4d6a-b41f-bd51682b6913-kube-api-access-jkpsl\") pod \"maas-api-revoke-sub-adf671fc-8zgws\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:38.109184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:38.109141    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d265226-b376-4d80-98b6-0f56ce3055cc" path="/var/lib/kubelet/pods/4d265226-b376-4d80-98b6-0f56ce3055cc/volumes"
Oct 30 01:18:38.115817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:38.115794    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:38.424037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:38.423965    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"]
Oct 30 01:18:38.425194 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:18:38.425164    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc377c2_639b_4d6a_b41f_bd51682b6913.slice/crio-8d29f7386ac9cf7803384a6e3d1f8501dafd2d780850bfdaee07a08bdae921c6 WatchSource:0}: Error finding container 8d29f7386ac9cf7803384a6e3d1f8501dafd2d780850bfdaee07a08bdae921c6: Status 404 returned error can't find the container with id 8d29f7386ac9cf7803384a6e3d1f8501dafd2d780850bfdaee07a08bdae921c6
Oct 30 01:18:38.573150 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:38.573121    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:38.752132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:38.752096    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws" event={"ID":"7bc377c2-639b-4d6a-b41f-bd51682b6913","Type":"ContainerStarted","Data":"8d29f7386ac9cf7803384a6e3d1f8501dafd2d780850bfdaee07a08bdae921c6"}
Oct 30 01:18:38.752684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:38.752161    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8889-73b17f" podUID="75d995d3-375f-4320-bd40-dd8ca586bc8f" containerName="test-curl-8889-73b17f" containerID="cri-o://afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f" gracePeriod=2
Oct 30 01:18:39.761197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:39.761160    4576 generic.go:356] "Generic (PLEG): container finished" podID="7bc377c2-639b-4d6a-b41f-bd51682b6913" containerID="0939467825ef196fd208e7d7f3e1d5fa73929586438564b4f339e4a252912f7e" exitCode=0
Oct 30 01:18:39.761666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:39.761274    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws" event={"ID":"7bc377c2-639b-4d6a-b41f-bd51682b6913","Type":"ContainerDied","Data":"0939467825ef196fd208e7d7f3e1d5fa73929586438564b4f339e4a252912f7e"}
Oct 30 01:18:39.815776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:39.815744    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:40.167169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.167054    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8" podUID="824c9aa8-ff1d-415c-90bb-7f0ac251a934" containerName="payload-processing" containerID="cri-o://78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3" gracePeriod=25
Oct 30 01:18:40.187525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.187496    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:40.470695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.470670    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:18:40.628066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.628033    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824c9aa8-ff1d-415c-90bb-7f0ac251a934-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824c9aa8-ff1d-415c-90bb-7f0ac251a934-plugins-config-volume\") pod \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\" (UID: \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\") "
Oct 30 01:18:40.628297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.628104    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824c9aa8-ff1d-415c-90bb-7f0ac251a934-kube-api-access-gff46\" (UniqueName: \"kubernetes.io/projected/824c9aa8-ff1d-415c-90bb-7f0ac251a934-kube-api-access-gff46\") pod \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\" (UID: \"824c9aa8-ff1d-415c-90bb-7f0ac251a934\") "
Oct 30 01:18:40.628511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.628485    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824c9aa8-ff1d-415c-90bb-7f0ac251a934-plugins-config-volume" pod "824c9aa8-ff1d-415c-90bb-7f0ac251a934" (UID: "824c9aa8-ff1d-415c-90bb-7f0ac251a934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:40.628659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.628644    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824c9aa8-ff1d-415c-90bb-7f0ac251a934-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:40.630572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.630546    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824c9aa8-ff1d-415c-90bb-7f0ac251a934-kube-api-access-gff46" pod "824c9aa8-ff1d-415c-90bb-7f0ac251a934" (UID: "824c9aa8-ff1d-415c-90bb-7f0ac251a934"). InnerVolumeSpecName "kube-api-access-gff46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:40.730830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.730740    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gff46\" (UniqueName: \"kubernetes.io/projected/824c9aa8-ff1d-415c-90bb-7f0ac251a934-kube-api-access-gff46\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:40.790813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.790774    4576 generic.go:356] "Generic (PLEG): container finished" podID="824c9aa8-ff1d-415c-90bb-7f0ac251a934" containerID="78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3" exitCode=0
Oct 30 01:18:40.791357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.791154    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"
Oct 30 01:18:40.791357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.791269    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8" event={"ID":"824c9aa8-ff1d-415c-90bb-7f0ac251a934","Type":"ContainerDied","Data":"78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3"}
Oct 30 01:18:40.791357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.791340    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8" event={"ID":"824c9aa8-ff1d-415c-90bb-7f0ac251a934","Type":"ContainerDied","Data":"9b8d9cfd20ad7c67849e64f440ea5457e52502d8a2ba29cbbcf369a46a0512b2"}
Oct 30 01:18:40.791557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.791369    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"]
Oct 30 01:18:40.791626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.791616    4576 scope.go:122] "RemoveContainer" containerID="78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3"
Oct 30 01:18:40.830963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.830938    4576 scope.go:122] "RemoveContainer" containerID="78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3"
Oct 30 01:18:40.831467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.831444    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:40.831765 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:18:40.831736    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3\": container with ID starting with 78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3 not found: ID does not exist" containerID="78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3"
Oct 30 01:18:40.831925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.831772    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3"} err="failed to get container status \"78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3\": rpc error: code = NotFound desc = could not find container \"78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3\": container with ID starting with 78e28469a6b9250256cd9b8604e6b99fe55abbb7704694224d3ed098ec9c97a3 not found: ID does not exist"
Oct 30 01:18:40.906436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.898718    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"]
Oct 30 01:18:40.910270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:40.909887    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1075cb-7757f48c64-l8qv8"]
Oct 30 01:18:41.058286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.057511    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:41.126050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.125869    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:41.148307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.139775    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc377c2-639b-4d6a-b41f-bd51682b6913-kube-api-access-jkpsl\" (UniqueName: \"kubernetes.io/projected/7bc377c2-639b-4d6a-b41f-bd51682b6913-kube-api-access-jkpsl\") pod \"7bc377c2-639b-4d6a-b41f-bd51682b6913\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") "
Oct 30 01:18:41.148307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.139858    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-xks-ca\") pod \"7bc377c2-639b-4d6a-b41f-bd51682b6913\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") "
Oct 30 01:18:41.148307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.139961    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-openshift-ca\") pod \"7bc377c2-639b-4d6a-b41f-bd51682b6913\" (UID: \"7bc377c2-639b-4d6a-b41f-bd51682b6913\") "
Oct 30 01:18:41.148307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.147645    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-openshift-ca" pod "7bc377c2-639b-4d6a-b41f-bd51682b6913" (UID: "7bc377c2-639b-4d6a-b41f-bd51682b6913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:41.158414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.158374    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-xks-ca" pod "7bc377c2-639b-4d6a-b41f-bd51682b6913" (UID: "7bc377c2-639b-4d6a-b41f-bd51682b6913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:41.159729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.159705    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc377c2-639b-4d6a-b41f-bd51682b6913-kube-api-access-jkpsl" pod "7bc377c2-639b-4d6a-b41f-bd51682b6913" (UID: "7bc377c2-639b-4d6a-b41f-bd51682b6913"). InnerVolumeSpecName "kube-api-access-jkpsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:41.241929 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.241887    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d995d3-375f-4320-bd40-dd8ca586bc8f-kube-api-access-bnv5j\" (UniqueName: \"kubernetes.io/projected/75d995d3-375f-4320-bd40-dd8ca586bc8f-kube-api-access-bnv5j\") pod \"75d995d3-375f-4320-bd40-dd8ca586bc8f\" (UID: \"75d995d3-375f-4320-bd40-dd8ca586bc8f\") "
Oct 30 01:18:41.242442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.242420    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpsl\" (UniqueName: \"kubernetes.io/projected/7bc377c2-639b-4d6a-b41f-bd51682b6913-kube-api-access-jkpsl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:41.242442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.242440    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:41.242442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.242450    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc377c2-639b-4d6a-b41f-bd51682b6913-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:41.244568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.244539    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d995d3-375f-4320-bd40-dd8ca586bc8f-kube-api-access-bnv5j" pod "75d995d3-375f-4320-bd40-dd8ca586bc8f" (UID: "75d995d3-375f-4320-bd40-dd8ca586bc8f"). InnerVolumeSpecName "kube-api-access-bnv5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:41.322380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.322303    4576 scope.go:122] "RemoveContainer" containerID="c9ebb32708f2fb9ec3baac5eb99bc2d994fc04ea1fcafca6c1d9b77fae37ddc2"
Oct 30 01:18:41.333607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.333545    4576 scope.go:122] "RemoveContainer" containerID="a57a61620b4dfe51f6e59b35f276df6ba884a1052ee29a7a6c921787335240d0"
Oct 30 01:18:41.343578 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.343535    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnv5j\" (UniqueName: \"kubernetes.io/projected/75d995d3-375f-4320-bd40-dd8ca586bc8f-kube-api-access-bnv5j\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:41.344805 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.344786    4576 scope.go:122] "RemoveContainer" containerID="18c5969c5c684ec1ccb6459fc04228fd94d4201d780cc57b1b444532a89521a8"
Oct 30 01:18:41.357031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.356990    4576 scope.go:122] "RemoveContainer" containerID="16886c5c35ab72cea42ee2a0265b6d9defc68f48e0ebbbf9d6c692143888a7c7"
Oct 30 01:18:41.369092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.369071    4576 scope.go:122] "RemoveContainer" containerID="349ba32179b62e5eedea71a6634768ab1c5958f16333ddf5832639791e19b9fe"
Oct 30 01:18:41.379327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.379307    4576 scope.go:122] "RemoveContainer" containerID="6c97fe38217c1d8ebe6da2bab1303a5ddec09cfe85810d809f38e4e65892b7a4"
Oct 30 01:18:41.388642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.388621    4576 scope.go:122] "RemoveContainer" containerID="f18669579f0064f68e1d91ca5a7c59d6ffaff0d24e32d307ee7e177913a5761b"
Oct 30 01:18:41.398859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.398841    4576 scope.go:122] "RemoveContainer" containerID="b046da7d69ebd29ae4412bd6f9d0f01fcbbc536a70148a5cda3d1ed8730bbca7"
Oct 30 01:18:41.460567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.460544    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:41.801125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.801085    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws" event={"ID":"7bc377c2-639b-4d6a-b41f-bd51682b6913","Type":"ContainerDied","Data":"8d29f7386ac9cf7803384a6e3d1f8501dafd2d780850bfdaee07a08bdae921c6"}
Oct 30 01:18:41.801125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.801116    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"
Oct 30 01:18:41.801706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.801144    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d29f7386ac9cf7803384a6e3d1f8501dafd2d780850bfdaee07a08bdae921c6"
Oct 30 01:18:41.803147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.803118    4576 generic.go:356] "Generic (PLEG): container finished" podID="75d995d3-375f-4320-bd40-dd8ca586bc8f" containerID="afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f" exitCode=137
Oct 30 01:18:41.803346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.803188    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:41.803637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.803519    4576 scope.go:122] "RemoveContainer" containerID="afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f"
Oct 30 01:18:41.823485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.823462    4576 scope.go:122] "RemoveContainer" containerID="afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f"
Oct 30 01:18:41.824057 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:18:41.823868    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f\": container with ID starting with afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f not found: ID does not exist" containerID="afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f"
Oct 30 01:18:41.824057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.823911    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f"} err="failed to get container status \"afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f\": rpc error: code = NotFound desc = could not find container \"afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f\": container with ID starting with afeab2adc6e730b1516a3153db28bbe6758e993c0db1920fa143eb98adedcd1f not found: ID does not exist"
Oct 30 01:18:41.835256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.833172    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"]
Oct 30 01:18:41.835256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.834280    4576 cpu_manager.go:414] "containerMap: removing container" podUID="7bc377c2-639b-4d6a-b41f-bd51682b6913" containerName="revoke-keys"
Oct 30 01:18:41.835256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.834300    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc377c2-639b-4d6a-b41f-bd51682b6913" containerName="revoke-keys"
Oct 30 01:18:41.835256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.834329    4576 cpu_manager.go:414] "containerMap: removing container" podUID="824c9aa8-ff1d-415c-90bb-7f0ac251a934" containerName="payload-processing"
Oct 30 01:18:41.835256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.834337    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824c9aa8-ff1d-415c-90bb-7f0ac251a934" containerName="payload-processing"
Oct 30 01:18:41.835256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.834494    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc377c2-639b-4d6a-b41f-bd51682b6913" containerName="revoke-keys"
Oct 30 01:18:41.835256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.834510    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="824c9aa8-ff1d-415c-90bb-7f0ac251a934" containerName="payload-processing"
Oct 30 01:18:41.845611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.845583    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.854635 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:18:41.854601    4576 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8889-73b17f\" is forbidden: User \"system:node:ip-10-0-130-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-5.ec2.internal' and this object" podUID="75d995d3-375f-4320-bd40-dd8ca586bc8f" pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:41.865494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.865404    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"]
Oct 30 01:18:41.872946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.867064    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5c515e-openshift-default-dockercfg-cqchw\""
Oct 30 01:18:41.887906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.887854    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:41.941262 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:18:41.937616    4576 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8889-73b17f\" is forbidden: User \"system:node:ip-10-0-130-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-5.ec2.internal' and this object" podUID="75d995d3-375f-4320-bd40-dd8ca586bc8f" pod="opendatahub/test-curl-8889-73b17f"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.953427    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hqk\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-kube-api-access-v4hqk\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.953718    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.953769    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.953848    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.953872    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.953954    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.953998    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.954031    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:41.958260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:41.954059    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.042301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.029924    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.057595    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.057670    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.057709    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.057745    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.057858    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hqk\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-kube-api-access-v4hqk\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.057969    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.058007    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.058095    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.058860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.058128    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.062176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.060318    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.062176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.060654    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.062176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.060901    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.062176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.061858    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.062816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.062793    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.095970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.095863    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hqk\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-kube-api-access-v4hqk\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.107596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.101569    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.108776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.108381    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.109421 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.109389    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.167849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.167812    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d995d3-375f-4320-bd40-dd8ca586bc8f" path="/var/lib/kubelet/pods/75d995d3-375f-4320-bd40-dd8ca586bc8f/volumes"
Oct 30 01:18:42.168396 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.168376    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824c9aa8-ff1d-415c-90bb-7f0ac251a934" path="/var/lib/kubelet/pods/824c9aa8-ff1d-415c-90bb-7f0ac251a934/volumes"
Oct 30 01:18:42.178529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.178489    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:42.386855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.386747    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:42.511680 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:18:42.511652    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e867f5_4e74_42b1_941b_8d5e8e6b56d5.slice/crio-ae6db0d692ca24383fc12c3503f88cafaaab2b17469c52e06f70d63c6a76e833 WatchSource:0}: Error finding container ae6db0d692ca24383fc12c3503f88cafaaab2b17469c52e06f70d63c6a76e833: Status 404 returned error can't find the container with id ae6db0d692ca24383fc12c3503f88cafaaab2b17469c52e06f70d63c6a76e833
Oct 30 01:18:42.512785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.512752    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"]
Oct 30 01:18:42.515137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.515098    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:18:42.515215 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.515189    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:18:42.515285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.515256    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:18:42.814742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.814701    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" event={"ID":"81e867f5-4e74-42b1-941b-8d5e8e6b56d5","Type":"ContainerStarted","Data":"0d372a03d8c6df1584fb849e279e326bdad9739794af16b76d2478d28302d10e"}
Oct 30 01:18:42.815249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.814769    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" event={"ID":"81e867f5-4e74-42b1-941b-8d5e8e6b56d5","Type":"ContainerStarted","Data":"ae6db0d692ca24383fc12c3503f88cafaaab2b17469c52e06f70d63c6a76e833"}
Oct 30 01:18:42.867572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:42.867508    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" podStartSLOduration=1.867486167 podStartE2EDuration="1.867486167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:42.860758045 +0000 UTC m=+2402.932081893" watchObservedRunningTime="2026-10-30 01:18:42.867486167 +0000 UTC m=+2402.938810003"
Oct 30 01:18:43.178913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:43.178826    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:43.185644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:43.185613    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:43.407398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:43.407350    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:43.683195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:43.683158    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:43.825077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:43.825050    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:43.826005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:43.825988    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:18:44.165248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:44.165195    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:45.325940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:45.325881    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:45.462023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:45.461982    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:45.581459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:45.581364    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:47.147226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.147190    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:47.270084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.270044    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:47.390384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.390353    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"]
Oct 30 01:18:47.395008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.394985    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:47.413321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.413296    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"]
Oct 30 01:18:47.451267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.443911    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5c515e\""
Oct 30 01:18:47.508773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.478479    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5c515e-dockercfg-ssrfw\""
Oct 30 01:18:47.508773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.480782    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d26335-e3da-4e79-ac32-1ca6851556ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb\" (UID: \"83d26335-e3da-4e79-ac32-1ca6851556ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:47.509019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.508855    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wz5\" (UniqueName: \"kubernetes.io/projected/83d26335-e3da-4e79-ac32-1ca6851556ee-kube-api-access-w7wz5\") pod \"payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb\" (UID: \"83d26335-e3da-4e79-ac32-1ca6851556ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:47.615731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.615600    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wz5\" (UniqueName: \"kubernetes.io/projected/83d26335-e3da-4e79-ac32-1ca6851556ee-kube-api-access-w7wz5\") pod \"payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb\" (UID: \"83d26335-e3da-4e79-ac32-1ca6851556ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:47.616284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.616223    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d26335-e3da-4e79-ac32-1ca6851556ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb\" (UID: \"83d26335-e3da-4e79-ac32-1ca6851556ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:47.617503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.617469    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d26335-e3da-4e79-ac32-1ca6851556ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb\" (UID: \"83d26335-e3da-4e79-ac32-1ca6851556ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:47.620098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.620071    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:47.643221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.643122    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wz5\" (UniqueName: \"kubernetes.io/projected/83d26335-e3da-4e79-ac32-1ca6851556ee-kube-api-access-w7wz5\") pod \"payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb\" (UID: \"83d26335-e3da-4e79-ac32-1ca6851556ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:47.709714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:47.709671    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:48.051777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.051748    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"]
Oct 30 01:18:48.052723 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:18:48.052693    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d26335_e3da_4e79_ac32_1ca6851556ee.slice/crio-bd39284e7063c20edde6a671e4f158ab954f826a7daf5afd5d2e3450838e17d1 WatchSource:0}: Error finding container bd39284e7063c20edde6a671e4f158ab954f826a7daf5afd5d2e3450838e17d1: Status 404 returned error can't find the container with id bd39284e7063c20edde6a671e4f158ab954f826a7daf5afd5d2e3450838e17d1
Oct 30 01:18:48.257591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.257553    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"]
Oct 30 01:18:48.262289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.262266    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.277981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.277955    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"]
Oct 30 01:18:48.278959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.278925    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:18:48.279119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.279103    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:18:48.326900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.326818    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.326900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.326858    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.326900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.326893    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz577\" (UniqueName: \"kubernetes.io/projected/1a40906a-fc26-4db0-9373-18a08bd8454e-kube-api-access-nz577\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.385242 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:18:48.385184    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:18:48.428303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.428264    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.428493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.428314    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.428493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.428368    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz577\" (UniqueName: \"kubernetes.io/projected/1a40906a-fc26-4db0-9373-18a08bd8454e-kube-api-access-nz577\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.429144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.429046    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.431050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.431026    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.448685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.448660    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz577\" (UniqueName: \"kubernetes.io/projected/1a40906a-fc26-4db0-9373-18a08bd8454e-kube-api-access-nz577\") pod \"maas-api-revoke-sub-8e4e5207-r9j4m\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.577981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.577897    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:48.862651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.862564    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb" event={"ID":"83d26335-e3da-4e79-ac32-1ca6851556ee","Type":"ContainerStarted","Data":"af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394"}
Oct 30 01:18:48.862651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.862645    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb" event={"ID":"83d26335-e3da-4e79-ac32-1ca6851556ee","Type":"ContainerStarted","Data":"bd39284e7063c20edde6a671e4f158ab954f826a7daf5afd5d2e3450838e17d1"}
Oct 30 01:18:48.862864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.862765    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:18:48.879213 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:18:48.879183    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a40906a_fc26_4db0_9373_18a08bd8454e.slice/crio-cf832ab1643ff9ec1f9c41d4a4664447ab5410c3907cc8c6b1580b394f79c737 WatchSource:0}: Error finding container cf832ab1643ff9ec1f9c41d4a4664447ab5410c3907cc8c6b1580b394f79c737: Status 404 returned error can't find the container with id cf832ab1643ff9ec1f9c41d4a4664447ab5410c3907cc8c6b1580b394f79c737
Oct 30 01:18:48.879612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.879589    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"]
Oct 30 01:18:48.902735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:48.902679    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb" podStartSLOduration=1.902659834 podStartE2EDuration="1.902659834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:48.902480943 +0000 UTC m=+2408.973804785" watchObservedRunningTime="2026-10-30 01:18:48.902659834 +0000 UTC m=+2408.973983671"
Oct 30 01:18:49.725303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:49.725271    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:49.872063 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:49.872027    4576 generic.go:356] "Generic (PLEG): container finished" podID="1a40906a-fc26-4db0-9373-18a08bd8454e" containerID="6951238c6d6181912a2ec116fc740f7463bb6a12901450c173aec2653cb5cb9d" exitCode=0
Oct 30 01:18:49.872273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:49.872107    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m" event={"ID":"1a40906a-fc26-4db0-9373-18a08bd8454e","Type":"ContainerDied","Data":"6951238c6d6181912a2ec116fc740f7463bb6a12901450c173aec2653cb5cb9d"}
Oct 30 01:18:49.872273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:49.872168    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m" event={"ID":"1a40906a-fc26-4db0-9373-18a08bd8454e","Type":"ContainerStarted","Data":"cf832ab1643ff9ec1f9c41d4a4664447ab5410c3907cc8c6b1580b394f79c737"}
Oct 30 01:18:51.071498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.071472    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:51.163816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.163624    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-xks-ca\") pod \"1a40906a-fc26-4db0-9373-18a08bd8454e\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") "
Oct 30 01:18:51.163816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.163724    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a40906a-fc26-4db0-9373-18a08bd8454e-kube-api-access-nz577\" (UniqueName: \"kubernetes.io/projected/1a40906a-fc26-4db0-9373-18a08bd8454e-kube-api-access-nz577\") pod \"1a40906a-fc26-4db0-9373-18a08bd8454e\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") "
Oct 30 01:18:51.164086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.163831    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-openshift-ca\") pod \"1a40906a-fc26-4db0-9373-18a08bd8454e\" (UID: \"1a40906a-fc26-4db0-9373-18a08bd8454e\") "
Oct 30 01:18:51.164637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.164603    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-openshift-ca" pod "1a40906a-fc26-4db0-9373-18a08bd8454e" (UID: "1a40906a-fc26-4db0-9373-18a08bd8454e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:51.166132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.166104    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a40906a-fc26-4db0-9373-18a08bd8454e-kube-api-access-nz577" pod "1a40906a-fc26-4db0-9373-18a08bd8454e" (UID: "1a40906a-fc26-4db0-9373-18a08bd8454e"). InnerVolumeSpecName "kube-api-access-nz577". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:51.166275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.166136    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-xks-ca" pod "1a40906a-fc26-4db0-9373-18a08bd8454e" (UID: "1a40906a-fc26-4db0-9373-18a08bd8454e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:51.271438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.271401    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.271438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.271435    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a40906a-fc26-4db0-9373-18a08bd8454e-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.271438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.271446    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz577\" (UniqueName: \"kubernetes.io/projected/1a40906a-fc26-4db0-9373-18a08bd8454e-kube-api-access-nz577\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.554177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.554131    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:51.889973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.889891    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m" event={"ID":"1a40906a-fc26-4db0-9373-18a08bd8454e","Type":"ContainerDied","Data":"cf832ab1643ff9ec1f9c41d4a4664447ab5410c3907cc8c6b1580b394f79c737"}
Oct 30 01:18:51.889973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.889915    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"
Oct 30 01:18:51.889973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:51.889967    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf832ab1643ff9ec1f9c41d4a4664447ab5410c3907cc8c6b1580b394f79c737"
Oct 30 01:18:53.045417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:53.045368    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:54.578598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:54.578565    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:54.887277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:54.887159    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"]
Oct 30 01:18:54.887934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:54.887905    4576 cpu_manager.go:414] "containerMap: removing container" podUID="1a40906a-fc26-4db0-9373-18a08bd8454e" containerName="revoke-keys"
Oct 30 01:18:54.887934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:54.887930    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a40906a-fc26-4db0-9373-18a08bd8454e" containerName="revoke-keys"
Oct 30 01:18:54.888153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:54.888071    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a40906a-fc26-4db0-9373-18a08bd8454e" containerName="revoke-keys"
Oct 30 01:18:54.892111 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:54.892090    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:54.925471 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:54.925434    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-ad00a0-openshift-default-dockercfg-w2ff8\""
Oct 30 01:18:54.929682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:54.929653    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"]
Oct 30 01:18:55.022116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022078    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-istio-token\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.022323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022198    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e17a171-80f1-445e-a664-b278014242c6-istio-podinfo\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.022323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022227    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-certs\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.022454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022333    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.022454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022366    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-envoy\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.022562 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022464    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-credential-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.022638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022620    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-data\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.022695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022681    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e17a171-80f1-445e-a664-b278014242c6-istiod-ca-cert\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.022730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.022711    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-kube-api-access-2qkq8\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.123968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.123929    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-data\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.123983    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e17a171-80f1-445e-a664-b278014242c6-istiod-ca-cert\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124001    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-kube-api-access-2qkq8\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124062    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-istio-token\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124096    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e17a171-80f1-445e-a664-b278014242c6-istio-podinfo\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124122    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-certs\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124163    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124198    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-envoy\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124273    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-credential-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124531    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-data\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.124870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.124840    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-credential-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.125187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.125063    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-socket\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.125292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.125204    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-certs\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.125292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.125263    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e17a171-80f1-445e-a664-b278014242c6-istiod-ca-cert\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.126926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.126900    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-envoy\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.127309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.127292    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e17a171-80f1-445e-a664-b278014242c6-istio-podinfo\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.145549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.145470    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-istio-token\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.145679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.145596    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-kube-api-access-2qkq8\") pod \"e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") " pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.206860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.206826    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:55.527845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.527814    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"]
Oct 30 01:18:55.528737 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:18:55.528704    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e17a171_80f1_445e_a664_b278014242c6.slice/crio-1c82d697b95eb84ee21e5f0b15b54093e1a07c444304ff22226c7b49afd31393 WatchSource:0}: Error finding container 1c82d697b95eb84ee21e5f0b15b54093e1a07c444304ff22226c7b49afd31393: Status 404 returned error can't find the container with id 1c82d697b95eb84ee21e5f0b15b54093e1a07c444304ff22226c7b49afd31393
Oct 30 01:18:55.531225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.531192    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:18:55.531314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.531288    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:18:55.531351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.531321    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:18:55.827333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.827188    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9" podUID="eb2e883e-babe-4ffd-bafb-369d08bc9b12" containerName="main" containerID="cri-o://56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac" gracePeriod=45
Oct 30 01:18:55.921390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.921355    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" event={"ID":"6e17a171-80f1-445e-a664-b278014242c6","Type":"ContainerStarted","Data":"11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b"}
Oct 30 01:18:55.921630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.921425    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" event={"ID":"6e17a171-80f1-445e-a664-b278014242c6","Type":"ContainerStarted","Data":"1c82d697b95eb84ee21e5f0b15b54093e1a07c444304ff22226c7b49afd31393"}
Oct 30 01:18:55.982271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:55.982206    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podStartSLOduration=1.9821918840000001 podStartE2EDuration="1.982191884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:55.980025187 +0000 UTC m=+2416.051349022" watchObservedRunningTime="2026-10-30 01:18:55.982191884 +0000 UTC m=+2416.053515718"
Oct 30 01:18:56.089319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.089294    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:18:56.139175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.139011    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb2e883e-babe-4ffd-bafb-369d08bc9b12-kube-api-access-fprml\" (UniqueName: \"kubernetes.io/projected/eb2e883e-babe-4ffd-bafb-369d08bc9b12-kube-api-access-fprml\") pod \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") "
Oct 30 01:18:56.139466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.139436    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tls-certs\") pod \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") "
Oct 30 01:18:56.139566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.139553    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tmp-dir\") pod \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") "
Oct 30 01:18:56.139634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.139617    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-model-cache\") pod \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") "
Oct 30 01:18:56.139723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.139708    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-dshm\") pod \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") "
Oct 30 01:18:56.139803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.139765    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-home\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-home\") pod \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\" (UID: \"eb2e883e-babe-4ffd-bafb-369d08bc9b12\") "
Oct 30 01:18:56.140723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.140692    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-home" pod "eb2e883e-babe-4ffd-bafb-369d08bc9b12" (UID: "eb2e883e-babe-4ffd-bafb-369d08bc9b12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:56.141492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.141323    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tmp-dir" pod "eb2e883e-babe-4ffd-bafb-369d08bc9b12" (UID: "eb2e883e-babe-4ffd-bafb-369d08bc9b12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:56.143074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.143048    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-model-cache" pod "eb2e883e-babe-4ffd-bafb-369d08bc9b12" (UID: "eb2e883e-babe-4ffd-bafb-369d08bc9b12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:56.144793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.144763    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2e883e-babe-4ffd-bafb-369d08bc9b12-kube-api-access-fprml" pod "eb2e883e-babe-4ffd-bafb-369d08bc9b12" (UID: "eb2e883e-babe-4ffd-bafb-369d08bc9b12"). InnerVolumeSpecName "kube-api-access-fprml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:56.145800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.145559    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tls-certs" pod "eb2e883e-babe-4ffd-bafb-369d08bc9b12" (UID: "eb2e883e-babe-4ffd-bafb-369d08bc9b12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:56.146285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.146084    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-dshm" pod "eb2e883e-babe-4ffd-bafb-369d08bc9b12" (UID: "eb2e883e-babe-4ffd-bafb-369d08bc9b12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:56.207251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.207202    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:18:56.208830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.208803    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:18:56.208957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.208864    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 01:18:56.241452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.241406    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:56.241452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.241443    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:56.241452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.241452    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:56.241452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.241462    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fprml\" (UniqueName: \"kubernetes.io/projected/eb2e883e-babe-4ffd-bafb-369d08bc9b12-kube-api-access-fprml\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:56.241755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.241472    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:56.241755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.241481    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2e883e-babe-4ffd-bafb-369d08bc9b12-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:18:56.284507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.284464    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:56.930836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.930798    4576 generic.go:356] "Generic (PLEG): container finished" podID="eb2e883e-babe-4ffd-bafb-369d08bc9b12" containerID="56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac" exitCode=0
Oct 30 01:18:56.931332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.930846    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"
Oct 30 01:18:56.931332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.930887    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9" event={"ID":"eb2e883e-babe-4ffd-bafb-369d08bc9b12","Type":"ContainerDied","Data":"56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac"}
Oct 30 01:18:56.931332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.930933    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9" event={"ID":"eb2e883e-babe-4ffd-bafb-369d08bc9b12","Type":"ContainerDied","Data":"d1a220bc4986564266046d76ee3c8c78f5f7134aad86f7f5986e66eeab10fa8e"}
Oct 30 01:18:56.931332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.930954    4576 scope.go:122] "RemoveContainer" containerID="56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac"
Oct 30 01:18:56.942256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.942221    4576 scope.go:122] "RemoveContainer" containerID="56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac"
Oct 30 01:18:56.942572 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:18:56.942551    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac\": container with ID starting with 56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac not found: ID does not exist" containerID="56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac"
Oct 30 01:18:56.942660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:56.942579    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac"} err="failed to get container status \"56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac\": rpc error: code = NotFound desc = could not find container \"56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac\": container with ID starting with 56e5dff812c22d6b3c6f2116c5439bd3dd03326b43eea110099d9e3071adb0ac not found: ID does not exist"
Oct 30 01:18:57.003351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:57.003319    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"]
Oct 30 01:18:57.020367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:57.019531    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-d0ae0d/rate-test-model-d0ae0d-kserve-856fb476f-hcwc9"]
Oct 30 01:18:57.208175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:57.208144    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:18:57.208377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:57.208196    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 01:18:57.819921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:57.819886    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:58.110475 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:58.110396    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb2e883e-babe-4ffd-bafb-369d08bc9b12" path="/var/lib/kubelet/pods/eb2e883e-babe-4ffd-bafb-369d08bc9b12/volumes"
Oct 30 01:18:58.208058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:58.208023    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:18:58.208321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:58.208080    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 01:18:59.207950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:59.207915    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:18:59.208385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:59.207965    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 01:18:59.492373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:59.486937    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:18:59.893081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:18:59.892990    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:19:00.208169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:00.208130    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:19:00.208636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:00.208258    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 01:19:00.821456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:00.821423    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:19:01.207300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:01.207268    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:19:01.207506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:01.207327    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 01:19:01.327107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:01.327075    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:02.209470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:02.209429    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 01:19:02.209692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:02.209512    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 01:19:03.213287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:03.213252    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:19:03.213806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:03.213758    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:19:03.214802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:03.214775    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:19:03.827013 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:03.826975    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:04.290097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:04.290065    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"]
Oct 30 01:19:04.334401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:04.334371    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:05.786769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:05.786740    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:05.896436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:05.896401    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:06.244731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.244692    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:06.715637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.715599    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"]
Oct 30 01:19:06.716629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.716604    4576 cpu_manager.go:414] "containerMap: removing container" podUID="eb2e883e-babe-4ffd-bafb-369d08bc9b12" containerName="main"
Oct 30 01:19:06.716772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.716756    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb2e883e-babe-4ffd-bafb-369d08bc9b12" containerName="main"
Oct 30 01:19:06.716982 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.716967    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb2e883e-babe-4ffd-bafb-369d08bc9b12" containerName="main"
Oct 30 01:19:06.720689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.720669    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.767715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.767682    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-954bf6-openshift-default-dockercfg-9c7x5\""
Oct 30 01:19:06.767969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.767741    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"]
Oct 30 01:19:06.790725 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.790678    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:06.837804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.837753    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-credential-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.838007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.837811    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.838007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.837929    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-token\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.838007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.837981    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6c4b4-b01f-4636-8244-26a809e97b77-istiod-ca-cert\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.838201 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.838090    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-certs\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.838201 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.838190    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-envoy\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.838645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.838270    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-data\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.838645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.838312    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-podinfo\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.838645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.838384    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngb5\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-kube-api-access-kngb5\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940325    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-envoy\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940404    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-data\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940452    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-podinfo\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940536    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngb5\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-kube-api-access-kngb5\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940584    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-credential-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940616    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940728    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-token\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940780    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6c4b4-b01f-4636-8244-26a809e97b77-istiod-ca-cert\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.940941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.940861    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-certs\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.941275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.941247    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-certs\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.941593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.941527    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-data\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.941593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.941548    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-credential-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.942143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.942091    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-socket\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.942576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.942543    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6c4b4-b01f-4636-8244-26a809e97b77-istiod-ca-cert\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.944730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.944703    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-envoy\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.947897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.947876    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-podinfo\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.963888 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.963861    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngb5\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-kube-api-access-kngb5\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:06.964279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:06.964201    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-token\") pod \"e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") " pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:07.036172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.036071    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:07.048383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.048326    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:07.255086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.253690    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8883-8c5452"]
Oct 30 01:19:07.260787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.260529    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8883-8c5452"
Oct 30 01:19:07.275150 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.275118    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8883-8c5452"]
Oct 30 01:19:07.280295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.280272    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-98k27\""
Oct 30 01:19:07.314457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.314369    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:07.356810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.356768    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx96h\" (UniqueName: \"kubernetes.io/projected/87b7494d-0eb1-499a-878f-eeef9047160e-kube-api-access-cx96h\") pod \"test-curl-8883-8c5452\" (UID: \"87b7494d-0eb1-499a-878f-eeef9047160e\") " pod="opendatahub/test-curl-8883-8c5452"
Oct 30 01:19:07.391787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.391759    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"]
Oct 30 01:19:07.394420 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:07.394388    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde6c4b4_b01f_4636_8244_26a809e97b77.slice/crio-d3edfcc8bb39ec3ff192a8317208692e3f9ada44156a04a92f31b27e27a4bcd3 WatchSource:0}: Error finding container d3edfcc8bb39ec3ff192a8317208692e3f9ada44156a04a92f31b27e27a4bcd3: Status 404 returned error can't find the container with id d3edfcc8bb39ec3ff192a8317208692e3f9ada44156a04a92f31b27e27a4bcd3
Oct 30 01:19:07.398436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.398402    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:07.398550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.398476    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:07.398550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.398507    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:07.462096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.462062    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx96h\" (UniqueName: \"kubernetes.io/projected/87b7494d-0eb1-499a-878f-eeef9047160e-kube-api-access-cx96h\") pod \"test-curl-8883-8c5452\" (UID: \"87b7494d-0eb1-499a-878f-eeef9047160e\") " pod="opendatahub/test-curl-8883-8c5452"
Oct 30 01:19:07.482721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.482682    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:07.485000 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.484971    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx96h\" (UniqueName: \"kubernetes.io/projected/87b7494d-0eb1-499a-878f-eeef9047160e-kube-api-access-cx96h\") pod \"test-curl-8883-8c5452\" (UID: \"87b7494d-0eb1-499a-878f-eeef9047160e\") " pod="opendatahub/test-curl-8883-8c5452"
Oct 30 01:19:07.579845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:07.579753    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8883-8c5452"
Oct 30 01:19:08.074101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:08.074057    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" event={"ID":"dde6c4b4-b01f-4636-8244-26a809e97b77","Type":"ContainerStarted","Data":"84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72"}
Oct 30 01:19:08.074577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:08.074163    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" event={"ID":"dde6c4b4-b01f-4636-8244-26a809e97b77","Type":"ContainerStarted","Data":"d3edfcc8bb39ec3ff192a8317208692e3f9ada44156a04a92f31b27e27a4bcd3"}
Oct 30 01:19:08.123359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:08.123326    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8883-8c5452"]
Oct 30 01:19:08.125541 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:08.125505    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87b7494d_0eb1_499a_878f_eeef9047160e.slice/crio-ef03d20e77595c202fde67bafae68820af83b70cddadc3b8354fe37efccf58b3 WatchSource:0}: Error finding container ef03d20e77595c202fde67bafae68820af83b70cddadc3b8354fe37efccf58b3: Status 404 returned error can't find the container with id ef03d20e77595c202fde67bafae68820af83b70cddadc3b8354fe37efccf58b3
Oct 30 01:19:08.145806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:08.145760    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" podStartSLOduration=2.145746844 podStartE2EDuration="2.145746844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:08.135677887 +0000 UTC m=+2428.207001723" watchObservedRunningTime="2026-10-30 01:19:08.145746844 +0000 UTC m=+2428.217070679"
Oct 30 01:19:08.479621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:08.479577    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:09.048607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.048576    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:09.055060 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.055035    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:09.084846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.084799    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8883-8c5452" event={"ID":"87b7494d-0eb1-499a-878f-eeef9047160e","Type":"ContainerStarted","Data":"4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c"}
Oct 30 01:19:09.085345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.084855    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8883-8c5452" event={"ID":"87b7494d-0eb1-499a-878f-eeef9047160e","Type":"ContainerStarted","Data":"ef03d20e77595c202fde67bafae68820af83b70cddadc3b8354fe37efccf58b3"}
Oct 30 01:19:09.085345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.085211    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:09.086340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.086315    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:19:09.156383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.156331    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8883-8c5452" podStartSLOduration=2.156317878 podStartE2EDuration="2.156317878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:09.153746926 +0000 UTC m=+2429.225070762" watchObservedRunningTime="2026-10-30 01:19:09.156317878 +0000 UTC m=+2429.227641712"
Oct 30 01:19:09.990728 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.990695    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8883-8c5452"]
Oct 30 01:19:09.995079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:09.995051    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8883-8c5452"]
Oct 30 01:19:10.140159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.140118    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8883-fbaa72"]
Oct 30 01:19:10.142442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.142414    4576 cpu_manager.go:414] "containerMap: removing container" podUID="87b7494d-0eb1-499a-878f-eeef9047160e" containerName="test-curl-8883-8c5452"
Oct 30 01:19:10.142442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.142441    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b7494d-0eb1-499a-878f-eeef9047160e" containerName="test-curl-8883-8c5452"
Oct 30 01:19:10.142804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.142788    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b7494d-0eb1-499a-878f-eeef9047160e" containerName="test-curl-8883-8c5452"
Oct 30 01:19:10.148269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.148248    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8883-fbaa72"
Oct 30 01:19:10.160209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.160184    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8883-fbaa72"]
Oct 30 01:19:10.202618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.202575    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:10.270671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.270560    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbd2d\" (UniqueName: \"kubernetes.io/projected/06728135-d86e-4573-8cc3-644e0c79cc2d-kube-api-access-gbd2d\") pod \"test-curl-8883-fbaa72\" (UID: \"06728135-d86e-4573-8cc3-644e0c79cc2d\") " pod="opendatahub/test-curl-8883-fbaa72"
Oct 30 01:19:10.371996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.371966    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbd2d\" (UniqueName: \"kubernetes.io/projected/06728135-d86e-4573-8cc3-644e0c79cc2d-kube-api-access-gbd2d\") pod \"test-curl-8883-fbaa72\" (UID: \"06728135-d86e-4573-8cc3-644e0c79cc2d\") " pod="opendatahub/test-curl-8883-fbaa72"
Oct 30 01:19:10.392717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.392686    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbd2d\" (UniqueName: \"kubernetes.io/projected/06728135-d86e-4573-8cc3-644e0c79cc2d-kube-api-access-gbd2d\") pod \"test-curl-8883-fbaa72\" (UID: \"06728135-d86e-4573-8cc3-644e0c79cc2d\") " pod="opendatahub/test-curl-8883-fbaa72"
Oct 30 01:19:10.449935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.449896    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"]
Oct 30 01:19:10.454643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.454621    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.463822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.463797    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8883-fbaa72"
Oct 30 01:19:10.475922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.475896    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"]
Oct 30 01:19:10.484701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.484677    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-60f38f-openshift-default-dockercfg-tvpjg\""
Oct 30 01:19:10.563956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.563898    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575027    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575150    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575178    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njr6\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-kube-api-access-2njr6\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575215    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575309    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575370    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575433    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575484    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.575921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.575583    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.676738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.676700    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.676930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.676759    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.676930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.676799    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.676930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.676876    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.676956    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677052    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677078    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njr6\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-kube-api-access-2njr6\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677112    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677202    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677259    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677725 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677701    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677797    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677871    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.677969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.677763    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.679512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.679488    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.679871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.679853    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.683680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.683643    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:10.697666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.697635    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.698705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.698677    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njr6\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-kube-api-access-2njr6\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.772745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.772711    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:10.789711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:10.789684    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8883-fbaa72"]
Oct 30 01:19:10.790636 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:10.790616    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06728135_d86e_4573_8cc3_644e0c79cc2d.slice/crio-9a5e2e78351af3626fc2e69499bd5c58d8deda3925ac9cd12bf7227b4473fc87 WatchSource:0}: Error finding container 9a5e2e78351af3626fc2e69499bd5c58d8deda3925ac9cd12bf7227b4473fc87: Status 404 returned error can't find the container with id 9a5e2e78351af3626fc2e69499bd5c58d8deda3925ac9cd12bf7227b4473fc87
Oct 30 01:19:11.098785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.098697    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"]
Oct 30 01:19:11.103159 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:11.103126    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b3c72c_700f_4ffb_af37_5998bf4a12e7.slice/crio-56080fd7893d1386c09b3b6447ed1395b525a1406a4078ca2d723e97808bb128 WatchSource:0}: Error finding container 56080fd7893d1386c09b3b6447ed1395b525a1406a4078ca2d723e97808bb128: Status 404 returned error can't find the container with id 56080fd7893d1386c09b3b6447ed1395b525a1406a4078ca2d723e97808bb128
Oct 30 01:19:11.104435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.104409    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8883-fbaa72" event={"ID":"06728135-d86e-4573-8cc3-644e0c79cc2d","Type":"ContainerStarted","Data":"743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345"}
Oct 30 01:19:11.104557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.104450    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8883-fbaa72" event={"ID":"06728135-d86e-4573-8cc3-644e0c79cc2d","Type":"ContainerStarted","Data":"9a5e2e78351af3626fc2e69499bd5c58d8deda3925ac9cd12bf7227b4473fc87"}
Oct 30 01:19:11.104877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.104827    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8883-8c5452" podUID="87b7494d-0eb1-499a-878f-eeef9047160e" containerName="test-curl-8883-8c5452" containerID="cri-o://4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c" gracePeriod=2
Oct 30 01:19:11.106161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.106132    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:11.106321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.106195    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:11.106321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.106227    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:11.146728 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.146682    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8883-fbaa72" podStartSLOduration=1.14665866 podStartE2EDuration="1.14665866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:11.144969359 +0000 UTC m=+2431.216293195" watchObservedRunningTime="2026-10-30 01:19:11.14665866 +0000 UTC m=+2431.217982494"
Oct 30 01:19:11.952749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.952716    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8883-fbaa72"]
Oct 30 01:19:11.956582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:11.956551    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8883-fbaa72"]
Oct 30 01:19:12.113609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.113570    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" event={"ID":"69b3c72c-700f-4ffb-af37-5998bf4a12e7","Type":"ContainerStarted","Data":"363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0"}
Oct 30 01:19:12.113812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.113630    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" event={"ID":"69b3c72c-700f-4ffb-af37-5998bf4a12e7","Type":"ContainerStarted","Data":"56080fd7893d1386c09b3b6447ed1395b525a1406a4078ca2d723e97808bb128"}
Oct 30 01:19:12.158880 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.158825    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" podStartSLOduration=2.158809657 podStartE2EDuration="2.158809657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:12.156531962 +0000 UTC m=+2432.227855796" watchObservedRunningTime="2026-10-30 01:19:12.158809657 +0000 UTC m=+2432.230133491"
Oct 30 01:19:12.189258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.177056    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:12.311002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.310908    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:12.435933 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.435901    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:12.666806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.664315    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"]
Oct 30 01:19:12.666806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.665269    4576 cpu_manager.go:414] "containerMap: removing container" podUID="06728135-d86e-4573-8cc3-644e0c79cc2d" containerName="test-curl-8883-fbaa72"
Oct 30 01:19:12.666806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.665290    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06728135-d86e-4573-8cc3-644e0c79cc2d" containerName="test-curl-8883-fbaa72"
Oct 30 01:19:12.666806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.665492    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="06728135-d86e-4573-8cc3-644e0c79cc2d" containerName="test-curl-8883-fbaa72"
Oct 30 01:19:12.670367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.670341    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"]
Oct 30 01:19:12.675252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.671384    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.699043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.699014    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:19:12.754255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.730370    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:19:12.795974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.795945    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:12.796715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.796439    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 01:19:12.796917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.796896    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 01:19:12.859413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.859377    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.859642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.859496    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rbzv\" (UniqueName: \"kubernetes.io/projected/2c39a658-acf1-4334-8f1f-12fa46ce65a5-kube-api-access-4rbzv\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.859843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.859818    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.960880 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.960833    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.961106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.960915    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.961106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.960953    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rbzv\" (UniqueName: \"kubernetes.io/projected/2c39a658-acf1-4334-8f1f-12fa46ce65a5-kube-api-access-4rbzv\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.961731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.961706    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.963522 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.963500    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:12.998824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:12.998789    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rbzv\" (UniqueName: \"kubernetes.io/projected/2c39a658-acf1-4334-8f1f-12fa46ce65a5-kube-api-access-4rbzv\") pod \"maas-api-revoke-sub-7e424f59-txv5c\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:13.121156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.121112    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8883-fbaa72" podUID="06728135-d86e-4573-8cc3-644e0c79cc2d" containerName="test-curl-8883-fbaa72" containerID="cri-o://743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345" gracePeriod=2
Oct 30 01:19:13.131107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.131083    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:13.420184 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.420159    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8883-8c5452"
Oct 30 01:19:13.465144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.465109    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"]
Oct 30 01:19:13.465842 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:13.465819    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c39a658_acf1_4334_8f1f_12fa46ce65a5.slice/crio-f987cc04b26e89fa950ff9382af862bc3de53bd1649805eec4efcde749b82069 WatchSource:0}: Error finding container f987cc04b26e89fa950ff9382af862bc3de53bd1649805eec4efcde749b82069: Status 404 returned error can't find the container with id f987cc04b26e89fa950ff9382af862bc3de53bd1649805eec4efcde749b82069
Oct 30 01:19:13.497710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.497677    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b7494d-0eb1-499a-878f-eeef9047160e-kube-api-access-cx96h\" (UniqueName: \"kubernetes.io/projected/87b7494d-0eb1-499a-878f-eeef9047160e-kube-api-access-cx96h\") pod \"87b7494d-0eb1-499a-878f-eeef9047160e\" (UID: \"87b7494d-0eb1-499a-878f-eeef9047160e\") "
Oct 30 01:19:13.504482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.504449    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b7494d-0eb1-499a-878f-eeef9047160e-kube-api-access-cx96h" pod "87b7494d-0eb1-499a-878f-eeef9047160e" (UID: "87b7494d-0eb1-499a-878f-eeef9047160e"). InnerVolumeSpecName "kube-api-access-cx96h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:13.603156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.603119    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx96h\" (UniqueName: \"kubernetes.io/projected/87b7494d-0eb1-499a-878f-eeef9047160e-kube-api-access-cx96h\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:13.792485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.792386    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 01:19:13.792485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.792471    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 01:19:13.997968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:13.997773    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:14.109749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.109674    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b7494d-0eb1-499a-878f-eeef9047160e" path="/var/lib/kubelet/pods/87b7494d-0eb1-499a-878f-eeef9047160e/volumes"
Oct 30 01:19:14.140972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.140936    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c" event={"ID":"2c39a658-acf1-4334-8f1f-12fa46ce65a5","Type":"ContainerStarted","Data":"404ab372b0bf44f3dda2ee9bb8a0427bf70e3a4713288b140a8b67a2509698cf"}
Oct 30 01:19:14.141152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.140991    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c" event={"ID":"2c39a658-acf1-4334-8f1f-12fa46ce65a5","Type":"ContainerStarted","Data":"f987cc04b26e89fa950ff9382af862bc3de53bd1649805eec4efcde749b82069"}
Oct 30 01:19:14.143434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.143404    4576 generic.go:356] "Generic (PLEG): container finished" podID="87b7494d-0eb1-499a-878f-eeef9047160e" containerID="4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c" exitCode=137
Oct 30 01:19:14.144361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.144339    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8883-8c5452"
Oct 30 01:19:14.144730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.144715    4576 scope.go:122] "RemoveContainer" containerID="4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c"
Oct 30 01:19:14.155340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.155320    4576 scope.go:122] "RemoveContainer" containerID="4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c"
Oct 30 01:19:14.155627 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:19:14.155605    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c\": container with ID starting with 4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c not found: ID does not exist" containerID="4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c"
Oct 30 01:19:14.155696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.155640    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c"} err="failed to get container status \"4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c\": rpc error: code = NotFound desc = could not find container \"4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c\": container with ID starting with 4c89115c40f5199f38d49822871750654d1b6d1dc372e478ebd1b2a15fa0f45c not found: ID does not exist"
Oct 30 01:19:14.206262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.205208    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c" podStartSLOduration=2.205187331 podStartE2EDuration="2.205187331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:14.181728085 +0000 UTC m=+2434.253051932" watchObservedRunningTime="2026-10-30 01:19:14.205187331 +0000 UTC m=+2434.276511169"
Oct 30 01:19:14.787264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.785666    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 01:19:14.787264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:14.785731    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 01:19:15.152441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.152409    4576 generic.go:356] "Generic (PLEG): container finished" podID="2c39a658-acf1-4334-8f1f-12fa46ce65a5" containerID="404ab372b0bf44f3dda2ee9bb8a0427bf70e3a4713288b140a8b67a2509698cf" exitCode=0
Oct 30 01:19:15.152590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.152495    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c" event={"ID":"2c39a658-acf1-4334-8f1f-12fa46ce65a5","Type":"ContainerDied","Data":"404ab372b0bf44f3dda2ee9bb8a0427bf70e3a4713288b140a8b67a2509698cf"}
Oct 30 01:19:15.154604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.154579    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:15.407491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.355368    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:15.742164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.742134    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8883-fbaa72"
Oct 30 01:19:15.799221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.799181    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 01:19:15.799769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.799291    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 01:19:15.841669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.841638    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06728135-d86e-4573-8cc3-644e0c79cc2d-kube-api-access-gbd2d\" (UniqueName: \"kubernetes.io/projected/06728135-d86e-4573-8cc3-644e0c79cc2d-kube-api-access-gbd2d\") pod \"06728135-d86e-4573-8cc3-644e0c79cc2d\" (UID: \"06728135-d86e-4573-8cc3-644e0c79cc2d\") "
Oct 30 01:19:15.843985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.843951    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06728135-d86e-4573-8cc3-644e0c79cc2d-kube-api-access-gbd2d" pod "06728135-d86e-4573-8cc3-644e0c79cc2d" (UID: "06728135-d86e-4573-8cc3-644e0c79cc2d"). InnerVolumeSpecName "kube-api-access-gbd2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:15.942993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:15.942957    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbd2d\" (UniqueName: \"kubernetes.io/projected/06728135-d86e-4573-8cc3-644e0c79cc2d-kube-api-access-gbd2d\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.110205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.110114    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06728135-d86e-4573-8cc3-644e0c79cc2d" path="/var/lib/kubelet/pods/06728135-d86e-4573-8cc3-644e0c79cc2d/volumes"
Oct 30 01:19:16.162174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.162149    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8883-fbaa72"
Oct 30 01:19:16.162381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.162180    4576 scope.go:122] "RemoveContainer" containerID="743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345"
Oct 30 01:19:16.162381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.162104    4576 generic.go:356] "Generic (PLEG): container finished" podID="06728135-d86e-4573-8cc3-644e0c79cc2d" containerID="743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345" exitCode=137
Oct 30 01:19:16.209173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.209121    4576 scope.go:122] "RemoveContainer" containerID="743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345"
Oct 30 01:19:16.209612 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:19:16.209590    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345\": container with ID starting with 743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345 not found: ID does not exist" containerID="743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345"
Oct 30 01:19:16.209722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.209621    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345"} err="failed to get container status \"743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345\": rpc error: code = NotFound desc = could not find container \"743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345\": container with ID starting with 743af3a51c9b71ca82b6a14b99e2f0565d6bc46727ac2b701829ec47f3620345 not found: ID does not exist"
Oct 30 01:19:16.393869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.393813    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:16.419691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.418580    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:16.448263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.448208    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-xks-ca\") pod \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") "
Oct 30 01:19:16.448447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.448364    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c39a658-acf1-4334-8f1f-12fa46ce65a5-kube-api-access-4rbzv\" (UniqueName: \"kubernetes.io/projected/2c39a658-acf1-4334-8f1f-12fa46ce65a5-kube-api-access-4rbzv\") pod \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") "
Oct 30 01:19:16.448447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.448424    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-openshift-ca\") pod \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\" (UID: \"2c39a658-acf1-4334-8f1f-12fa46ce65a5\") "
Oct 30 01:19:16.448832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.448804    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-openshift-ca" pod "2c39a658-acf1-4334-8f1f-12fa46ce65a5" (UID: "2c39a658-acf1-4334-8f1f-12fa46ce65a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:16.449202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.449183    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.453265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.452384    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-xks-ca" pod "2c39a658-acf1-4334-8f1f-12fa46ce65a5" (UID: "2c39a658-acf1-4334-8f1f-12fa46ce65a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:16.453265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.452538    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c39a658-acf1-4334-8f1f-12fa46ce65a5-kube-api-access-4rbzv" pod "2c39a658-acf1-4334-8f1f-12fa46ce65a5" (UID: "2c39a658-acf1-4334-8f1f-12fa46ce65a5"). InnerVolumeSpecName "kube-api-access-4rbzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:16.550447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.550411    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rbzv\" (UniqueName: \"kubernetes.io/projected/2c39a658-acf1-4334-8f1f-12fa46ce65a5-kube-api-access-4rbzv\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.550447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.550441    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c39a658-acf1-4334-8f1f-12fa46ce65a5-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.790262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.789666    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 01:19:16.790262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:16.789728    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 01:19:17.205068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:17.205038    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:17.206682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:17.206664    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"
Oct 30 01:19:17.206938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:17.206674    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c" event={"ID":"2c39a658-acf1-4334-8f1f-12fa46ce65a5","Type":"ContainerDied","Data":"f987cc04b26e89fa950ff9382af862bc3de53bd1649805eec4efcde749b82069"}
Oct 30 01:19:17.207332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:17.207100    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f987cc04b26e89fa950ff9382af862bc3de53bd1649805eec4efcde749b82069"
Oct 30 01:19:17.905778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:17.905750    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:17.906298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:17.906276    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:17.907177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:17.907157    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:19:18.381675 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:19:18.381632    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:19:18.611358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:18.611316    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:19.313064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.313017    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2" podUID="2e4fc1ab-ec7a-4056-a97d-220d8527b083" containerName="main" containerID="cri-o://8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641" gracePeriod=45
Oct 30 01:19:19.616330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.616299    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:19:19.735431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.735392    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-model-cache\") pod \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") "
Oct 30 01:19:19.735630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.735600    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4fc1ab-ec7a-4056-a97d-220d8527b083-kube-api-access-7j4jg\" (UniqueName: \"kubernetes.io/projected/2e4fc1ab-ec7a-4056-a97d-220d8527b083-kube-api-access-7j4jg\") pod \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") "
Oct 30 01:19:19.735703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.735652    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tmp-dir\") pod \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") "
Oct 30 01:19:19.735703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.735675    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-model-cache" pod "2e4fc1ab-ec7a-4056-a97d-220d8527b083" (UID: "2e4fc1ab-ec7a-4056-a97d-220d8527b083"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:19.735792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.735729    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-dshm\") pod \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") "
Oct 30 01:19:19.735895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.735879    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tls-certs\") pod \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") "
Oct 30 01:19:19.735984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.735915    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-home\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-home\") pod \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\" (UID: \"2e4fc1ab-ec7a-4056-a97d-220d8527b083\") "
Oct 30 01:19:19.736050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.735984    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tmp-dir" pod "2e4fc1ab-ec7a-4056-a97d-220d8527b083" (UID: "2e4fc1ab-ec7a-4056-a97d-220d8527b083"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:19.736305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.736277    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-home" pod "2e4fc1ab-ec7a-4056-a97d-220d8527b083" (UID: "2e4fc1ab-ec7a-4056-a97d-220d8527b083"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:19.736740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.736716    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:19.736837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.736745    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:19.736837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.736759    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:19.738433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.738403    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4fc1ab-ec7a-4056-a97d-220d8527b083-kube-api-access-7j4jg" pod "2e4fc1ab-ec7a-4056-a97d-220d8527b083" (UID: "2e4fc1ab-ec7a-4056-a97d-220d8527b083"). InnerVolumeSpecName "kube-api-access-7j4jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:19.738535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.738449    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tls-certs" pod "2e4fc1ab-ec7a-4056-a97d-220d8527b083" (UID: "2e4fc1ab-ec7a-4056-a97d-220d8527b083"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:19.738535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.738469    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-dshm" pod "2e4fc1ab-ec7a-4056-a97d-220d8527b083" (UID: "2e4fc1ab-ec7a-4056-a97d-220d8527b083"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:19.838047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.837965    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j4jg\" (UniqueName: \"kubernetes.io/projected/2e4fc1ab-ec7a-4056-a97d-220d8527b083-kube-api-access-7j4jg\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:19.838047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.838014    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e4fc1ab-ec7a-4056-a97d-220d8527b083-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:19.838047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:19.838027    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e4fc1ab-ec7a-4056-a97d-220d8527b083-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:20.231400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.231368    4576 generic.go:356] "Generic (PLEG): container finished" podID="2e4fc1ab-ec7a-4056-a97d-220d8527b083" containerID="8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641" exitCode=0
Oct 30 01:19:20.231586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.231418    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"
Oct 30 01:19:20.231586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.231448    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2" event={"ID":"2e4fc1ab-ec7a-4056-a97d-220d8527b083","Type":"ContainerDied","Data":"8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641"}
Oct 30 01:19:20.231586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.231491    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2" event={"ID":"2e4fc1ab-ec7a-4056-a97d-220d8527b083","Type":"ContainerDied","Data":"b3f57627dec6164187b478d0502fb92d7a2a35453499c86ff2233213e2800655"}
Oct 30 01:19:20.231586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.231514    4576 scope.go:122] "RemoveContainer" containerID="8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641"
Oct 30 01:19:20.241787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.241765    4576 scope.go:122] "RemoveContainer" containerID="8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641"
Oct 30 01:19:20.242073 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:19:20.242052    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641\": container with ID starting with 8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641 not found: ID does not exist" containerID="8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641"
Oct 30 01:19:20.242139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.242081    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641"} err="failed to get container status \"8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641\": rpc error: code = NotFound desc = could not find container \"8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641\": container with ID starting with 8ff67f99a243de5bfe519772efcdfee8a3afdc1c9e0f20b0af12c6f3cb271641 not found: ID does not exist"
Oct 30 01:19:20.291002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.290972    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"]
Oct 30 01:19:20.296620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.296588    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/auth-test-model-24d5ce-kserve-7784766b97-gj2c2"]
Oct 30 01:19:20.334074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:20.334045    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:22.061431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.061389    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:22.147807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.146274    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4fc1ab-ec7a-4056-a97d-220d8527b083" path="/var/lib/kubelet/pods/2e4fc1ab-ec7a-4056-a97d-220d8527b083/volumes"
Oct 30 01:19:22.857632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.857596    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"]
Oct 30 01:19:22.858185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.858171    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2c39a658-acf1-4334-8f1f-12fa46ce65a5" containerName="revoke-keys"
Oct 30 01:19:22.858287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.858186    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c39a658-acf1-4334-8f1f-12fa46ce65a5" containerName="revoke-keys"
Oct 30 01:19:22.858287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.858223    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2e4fc1ab-ec7a-4056-a97d-220d8527b083" containerName="main"
Oct 30 01:19:22.858287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.858241    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4fc1ab-ec7a-4056-a97d-220d8527b083" containerName="main"
Oct 30 01:19:22.858396 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.858333    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4fc1ab-ec7a-4056-a97d-220d8527b083" containerName="main"
Oct 30 01:19:22.858396 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.858344    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c39a658-acf1-4334-8f1f-12fa46ce65a5" containerName="revoke-keys"
Oct 30 01:19:22.863425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.863407    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:22.884789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.884760    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"]
Oct 30 01:19:22.889662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.889633    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:19:22.889814 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.889642    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:19:22.981831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.981791    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:22.982033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.982017    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:22.982088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:22.982062    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqzb\" (UniqueName: \"kubernetes.io/projected/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-kube-api-access-tbqzb\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:23.083221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.083183    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:23.083700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.083356    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:23.083700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.083396    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqzb\" (UniqueName: \"kubernetes.io/projected/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-kube-api-access-tbqzb\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:23.084113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.084090    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:23.086495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.086470    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:23.104377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.104296    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqzb\" (UniqueName: \"kubernetes.io/projected/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-kube-api-access-tbqzb\") pod \"maas-api-revoke-sub-26d9a2c9-pf6rm\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:23.184716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.184639    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:23.487266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.487218    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"]
Oct 30 01:19:23.487822 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:23.487794    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c55f2bc_6e5e_46b3_88ad_70a48c525dfa.slice/crio-3cdcb9aa2372e32e9967e538fe7c387e3b8b26a7708a720e5d21572119be2b53 WatchSource:0}: Error finding container 3cdcb9aa2372e32e9967e538fe7c387e3b8b26a7708a720e5d21572119be2b53: Status 404 returned error can't find the container with id 3cdcb9aa2372e32e9967e538fe7c387e3b8b26a7708a720e5d21572119be2b53
Oct 30 01:19:23.771495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:23.771418    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:24.280622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:24.280586    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm" event={"ID":"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa","Type":"ContainerStarted","Data":"1d46387863e575d26a2bf00d684332a349947d8f27bbc367e5b5aeed2a7de5b8"}
Oct 30 01:19:24.281079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:24.280641    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm" event={"ID":"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa","Type":"ContainerStarted","Data":"3cdcb9aa2372e32e9967e538fe7c387e3b8b26a7708a720e5d21572119be2b53"}
Oct 30 01:19:24.324976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:24.324924    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm" podStartSLOduration=2.324908869 podStartE2EDuration="2.324908869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:24.32154613 +0000 UTC m=+2444.392869965" watchObservedRunningTime="2026-10-30 01:19:24.324908869 +0000 UTC m=+2444.396232705"
Oct 30 01:19:25.289470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:25.289435    4576 generic.go:356] "Generic (PLEG): container finished" podID="3c55f2bc-6e5e-46b3-88ad-70a48c525dfa" containerID="1d46387863e575d26a2bf00d684332a349947d8f27bbc367e5b5aeed2a7de5b8" exitCode=0
Oct 30 01:19:25.289978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:25.289473    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm" event={"ID":"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa","Type":"ContainerDied","Data":"1d46387863e575d26a2bf00d684332a349947d8f27bbc367e5b5aeed2a7de5b8"}
Oct 30 01:19:25.370610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:25.370577    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:25.664921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:25.664838    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:26.045900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.045864    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"]
Oct 30 01:19:26.066880 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.066844    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.070708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.070615    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"]
Oct 30 01:19:26.094647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.094618    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-07a3a8-openshift-default-dockercfg-xw9br\""
Oct 30 01:19:26.111281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.111248    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:26.236293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.233456    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffw4\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-kube-api-access-2ffw4\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.241367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.241333    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.246094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.246061    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.253382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.253331    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.255382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.255343    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.263968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.263460    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.264217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.264190    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289432fe-271c-4f17-8ae3-edc5b8794d46-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.264447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.264427    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.264694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.264676    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.269292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.268831    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:26.387260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.385159    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.389625    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.389697    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289432fe-271c-4f17-8ae3-edc5b8794d46-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.389798    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.389904    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.390051    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffw4\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-kube-api-access-2ffw4\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.390096    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.390166    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.390198    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.400391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.393568    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.415736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.413845    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289432fe-271c-4f17-8ae3-edc5b8794d46-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.415736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.414396    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.415736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.414626    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.415736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.414829    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.429491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.428564    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.429491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.429066    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.436718 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.436307    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.437853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.437789    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffw4\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-kube-api-access-2ffw4\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:26.583518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.583452    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:26.592985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.592955    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-kube-api-access-tbqzb\" (UniqueName: \"kubernetes.io/projected/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-kube-api-access-tbqzb\") pod \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") "
Oct 30 01:19:26.593093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.593037    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-openshift-ca\") pod \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") "
Oct 30 01:19:26.593093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.593085    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-xks-ca\") pod \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\" (UID: \"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa\") "
Oct 30 01:19:26.593475 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.593449    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-openshift-ca" pod "3c55f2bc-6e5e-46b3-88ad-70a48c525dfa" (UID: "3c55f2bc-6e5e-46b3-88ad-70a48c525dfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:26.593753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.593705    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:26.595409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.595349    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-kube-api-access-tbqzb" pod "3c55f2bc-6e5e-46b3-88ad-70a48c525dfa" (UID: "3c55f2bc-6e5e-46b3-88ad-70a48c525dfa"). InnerVolumeSpecName "kube-api-access-tbqzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:26.595690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.595671    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-xks-ca" pod "3c55f2bc-6e5e-46b3-88ad-70a48c525dfa" (UID: "3c55f2bc-6e5e-46b3-88ad-70a48c525dfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:26.695168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.695133    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbqzb\" (UniqueName: \"kubernetes.io/projected/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-kube-api-access-tbqzb\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:26.695168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.695164    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:26.697646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:26.697622    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:27.020187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.020160    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"]
Oct 30 01:19:27.020348 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:27.020185    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289432fe_271c_4f17_8ae3_edc5b8794d46.slice/crio-faeb82641062c1c1b3a5e6541a8a6d8717a36053434e0aa9f1ea0d78fb2c98a1 WatchSource:0}: Error finding container faeb82641062c1c1b3a5e6541a8a6d8717a36053434e0aa9f1ea0d78fb2c98a1: Status 404 returned error can't find the container with id faeb82641062c1c1b3a5e6541a8a6d8717a36053434e0aa9f1ea0d78fb2c98a1
Oct 30 01:19:27.023343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.023299    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:27.023447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.023381    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:27.023447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.023412    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:27.154640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.154607    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:27.313603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.313504    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"
Oct 30 01:19:27.313603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.313522    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm" event={"ID":"3c55f2bc-6e5e-46b3-88ad-70a48c525dfa","Type":"ContainerDied","Data":"3cdcb9aa2372e32e9967e538fe7c387e3b8b26a7708a720e5d21572119be2b53"}
Oct 30 01:19:27.313603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.313576    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cdcb9aa2372e32e9967e538fe7c387e3b8b26a7708a720e5d21572119be2b53"
Oct 30 01:19:27.315678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.315650    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" event={"ID":"289432fe-271c-4f17-8ae3-edc5b8794d46","Type":"ContainerStarted","Data":"c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f"}
Oct 30 01:19:27.315801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.315699    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" event={"ID":"289432fe-271c-4f17-8ae3-edc5b8794d46","Type":"ContainerStarted","Data":"faeb82641062c1c1b3a5e6541a8a6d8717a36053434e0aa9f1ea0d78fb2c98a1"}
Oct 30 01:19:27.329286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.329255    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:27.361105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.361053    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podStartSLOduration=1.361035108 podStartE2EDuration="1.361035108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:27.359135993 +0000 UTC m=+2447.430459829" watchObservedRunningTime="2026-10-30 01:19:27.361035108 +0000 UTC m=+2447.432358944"
Oct 30 01:19:27.698664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:27.698632    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:28.704766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:28.704723    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:28.715577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:28.715546    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:28.826054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:28.826020    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:28.939226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:28.939183    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:29.330408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:29.330379    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:29.331372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:29.331351    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:19:29.943619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:29.943582    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:30.113264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:30.098957    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:30.899520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:30.899478    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:31.828676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.828631    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:31.918059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.918026    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"]
Oct 30 01:19:31.918865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.918844    4576 cpu_manager.go:414] "containerMap: removing container" podUID="3c55f2bc-6e5e-46b3-88ad-70a48c525dfa" containerName="revoke-keys"
Oct 30 01:19:31.918967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.918865    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c55f2bc-6e5e-46b3-88ad-70a48c525dfa" containerName="revoke-keys"
Oct 30 01:19:31.919013 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.918969    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c55f2bc-6e5e-46b3-88ad-70a48c525dfa" containerName="revoke-keys"
Oct 30 01:19:31.922910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.922891    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:31.940374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.940346    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"]
Oct 30 01:19:31.944956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.944922    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:31.948572 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.948550    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-07a3a8\""
Oct 30 01:19:31.989133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.989103    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"]
Oct 30 01:19:31.993519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:31.993495    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:32.010635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.010514    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-07a3a8-dockercfg-m24r7\""
Oct 30 01:19:32.013697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.013670    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"]
Oct 30 01:19:32.101307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.094860    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f06dd1-fc1b-498a-8658-5639c6bd0951-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p\" (UID: \"93f06dd1-fc1b-498a-8658-5639c6bd0951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:32.101307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.094970    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc7xx\" (UniqueName: \"kubernetes.io/projected/93f06dd1-fc1b-498a-8658-5639c6bd0951-kube-api-access-cc7xx\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p\" (UID: \"93f06dd1-fc1b-498a-8658-5639c6bd0951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:32.101307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.095118    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbzjw\" (UniqueName: \"kubernetes.io/projected/07a9b46f-d980-4465-9dcf-158a3b2d9de0-kube-api-access-xbzjw\") pod \"payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb\" (UID: \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:32.101307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.095221    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a9b46f-d980-4465-9dcf-158a3b2d9de0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb\" (UID: \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:32.201291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.197505    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f06dd1-fc1b-498a-8658-5639c6bd0951-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p\" (UID: \"93f06dd1-fc1b-498a-8658-5639c6bd0951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:32.201291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.197593    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7xx\" (UniqueName: \"kubernetes.io/projected/93f06dd1-fc1b-498a-8658-5639c6bd0951-kube-api-access-cc7xx\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p\" (UID: \"93f06dd1-fc1b-498a-8658-5639c6bd0951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:32.201291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.197726    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbzjw\" (UniqueName: \"kubernetes.io/projected/07a9b46f-d980-4465-9dcf-158a3b2d9de0-kube-api-access-xbzjw\") pod \"payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb\" (UID: \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:32.201291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.197820    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a9b46f-d980-4465-9dcf-158a3b2d9de0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb\" (UID: \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:32.201291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.198766    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a9b46f-d980-4465-9dcf-158a3b2d9de0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb\" (UID: \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:32.201291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.199476    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f06dd1-fc1b-498a-8658-5639c6bd0951-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p\" (UID: \"93f06dd1-fc1b-498a-8658-5639c6bd0951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:32.227660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.223063    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbzjw\" (UniqueName: \"kubernetes.io/projected/07a9b46f-d980-4465-9dcf-158a3b2d9de0-kube-api-access-xbzjw\") pod \"payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb\" (UID: \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:32.227660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.225783    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7xx\" (UniqueName: \"kubernetes.io/projected/93f06dd1-fc1b-498a-8658-5639c6bd0951-kube-api-access-cc7xx\") pod \"payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p\" (UID: \"93f06dd1-fc1b-498a-8658-5639c6bd0951\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:32.239120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.239089    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:32.315044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.315011    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:32.550022 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:32.549992    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a9b46f_d980_4465_9dcf_158a3b2d9de0.slice/crio-3642d95933d92c90b3bd7c1d27b7f0f1d7393fa9a33561687d45711591b30a03 WatchSource:0}: Error finding container 3642d95933d92c90b3bd7c1d27b7f0f1d7393fa9a33561687d45711591b30a03: Status 404 returned error can't find the container with id 3642d95933d92c90b3bd7c1d27b7f0f1d7393fa9a33561687d45711591b30a03
Oct 30 01:19:32.550526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.550506    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"]
Oct 30 01:19:32.653582 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:32.653519    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f06dd1_fc1b_498a_8658_5639c6bd0951.slice/crio-1d6b45dcf5ffa67499e2eef4ee58e8044d50852ce89a3753760cdc1bfb8b9fa1 WatchSource:0}: Error finding container 1d6b45dcf5ffa67499e2eef4ee58e8044d50852ce89a3753760cdc1bfb8b9fa1: Status 404 returned error can't find the container with id 1d6b45dcf5ffa67499e2eef4ee58e8044d50852ce89a3753760cdc1bfb8b9fa1
Oct 30 01:19:32.654091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.654064    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"]
Oct 30 01:19:32.669362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:32.669336    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:33.381827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.381728    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p" event={"ID":"93f06dd1-fc1b-498a-8658-5639c6bd0951","Type":"ContainerStarted","Data":"3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4"}
Oct 30 01:19:33.381827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.381821    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p" event={"ID":"93f06dd1-fc1b-498a-8658-5639c6bd0951","Type":"ContainerStarted","Data":"1d6b45dcf5ffa67499e2eef4ee58e8044d50852ce89a3753760cdc1bfb8b9fa1"}
Oct 30 01:19:33.382416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.382059    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:33.383625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.383602    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb" event={"ID":"07a9b46f-d980-4465-9dcf-158a3b2d9de0","Type":"ContainerStarted","Data":"6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34"}
Oct 30 01:19:33.383762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.383636    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb" event={"ID":"07a9b46f-d980-4465-9dcf-158a3b2d9de0","Type":"ContainerStarted","Data":"3642d95933d92c90b3bd7c1d27b7f0f1d7393fa9a33561687d45711591b30a03"}
Oct 30 01:19:33.383762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.383723    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:33.425691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.425643    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p" podStartSLOduration=2.4256305080000002 podStartE2EDuration="2.425630508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:33.422365106 +0000 UTC m=+2453.493688954" watchObservedRunningTime="2026-10-30 01:19:33.425630508 +0000 UTC m=+2453.496954343"
Oct 30 01:19:33.464775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.464726    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb" podStartSLOduration=2.464712674 podStartE2EDuration="2.464712674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:33.463131087 +0000 UTC m=+2453.534454922" watchObservedRunningTime="2026-10-30 01:19:33.464712674 +0000 UTC m=+2453.536036509"
Oct 30 01:19:33.707134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.707098    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"]
Oct 30 01:19:33.711778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.711753    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.728682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.728652    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:19:33.729091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.729053    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"]
Oct 30 01:19:33.742518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.742486    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:19:33.828308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.828272    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c65765-23ce-437d-be07-4f136d09f263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.828468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.828389    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c65765-23ce-437d-be07-4f136d09f263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.828602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.828579    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt8kg\" (UniqueName: \"kubernetes.io/projected/32c65765-23ce-437d-be07-4f136d09f263-kube-api-access-qt8kg\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.929986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.929948    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c65765-23ce-437d-be07-4f136d09f263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.930187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.930019    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c65765-23ce-437d-be07-4f136d09f263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.930187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.930161    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt8kg\" (UniqueName: \"kubernetes.io/projected/32c65765-23ce-437d-be07-4f136d09f263-kube-api-access-qt8kg\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.930724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.930704    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c65765-23ce-437d-be07-4f136d09f263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.932494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.932475    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c65765-23ce-437d-be07-4f136d09f263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:33.950932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:33.950897    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt8kg\" (UniqueName: \"kubernetes.io/projected/32c65765-23ce-437d-be07-4f136d09f263-kube-api-access-qt8kg\") pod \"maas-api-revoke-sub-3b73df9b-5w2qz\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:34.027558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:34.027481    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:34.285446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:34.285360    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:34.340031 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:34.339988    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c65765_23ce_437d_be07_4f136d09f263.slice/crio-f9a25cb9bc5d83ebfabccd7d7f16d81dd6ae36d67ddd746128151caeef31fa99 WatchSource:0}: Error finding container f9a25cb9bc5d83ebfabccd7d7f16d81dd6ae36d67ddd746128151caeef31fa99: Status 404 returned error can't find the container with id f9a25cb9bc5d83ebfabccd7d7f16d81dd6ae36d67ddd746128151caeef31fa99
Oct 30 01:19:34.340600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:34.340580    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"]
Oct 30 01:19:34.396821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:34.396793    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz" event={"ID":"32c65765-23ce-437d-be07-4f136d09f263","Type":"ContainerStarted","Data":"f9a25cb9bc5d83ebfabccd7d7f16d81dd6ae36d67ddd746128151caeef31fa99"}
Oct 30 01:19:35.406333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:35.406303    4576 generic.go:356] "Generic (PLEG): container finished" podID="32c65765-23ce-437d-be07-4f136d09f263" containerID="0b25edceab866e882d324f0a7ca462c243a5c4f61288d4ced741e4abf6b3797e" exitCode=0
Oct 30 01:19:35.406754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:35.406345    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz" event={"ID":"32c65765-23ce-437d-be07-4f136d09f263","Type":"ContainerDied","Data":"0b25edceab866e882d324f0a7ca462c243a5c4f61288d4ced741e4abf6b3797e"}
Oct 30 01:19:35.827827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:35.827795    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:36.585803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.585775    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:36.770380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.770339    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c65765-23ce-437d-be07-4f136d09f263-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c65765-23ce-437d-be07-4f136d09f263-maas-api-xks-ca\") pod \"32c65765-23ce-437d-be07-4f136d09f263\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") "
Oct 30 01:19:36.770564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.770494    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32c65765-23ce-437d-be07-4f136d09f263-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c65765-23ce-437d-be07-4f136d09f263-maas-api-openshift-ca\") pod \"32c65765-23ce-437d-be07-4f136d09f263\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") "
Oct 30 01:19:36.770734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.770719    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c65765-23ce-437d-be07-4f136d09f263-kube-api-access-qt8kg\" (UniqueName: \"kubernetes.io/projected/32c65765-23ce-437d-be07-4f136d09f263-kube-api-access-qt8kg\") pod \"32c65765-23ce-437d-be07-4f136d09f263\" (UID: \"32c65765-23ce-437d-be07-4f136d09f263\") "
Oct 30 01:19:36.770971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.770944    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c65765-23ce-437d-be07-4f136d09f263-maas-api-openshift-ca" pod "32c65765-23ce-437d-be07-4f136d09f263" (UID: "32c65765-23ce-437d-be07-4f136d09f263"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:36.771226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.771208    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c65765-23ce-437d-be07-4f136d09f263-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:36.772728 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.772702    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c65765-23ce-437d-be07-4f136d09f263-maas-api-xks-ca" pod "32c65765-23ce-437d-be07-4f136d09f263" (UID: "32c65765-23ce-437d-be07-4f136d09f263"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:36.773203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.773186    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c65765-23ce-437d-be07-4f136d09f263-kube-api-access-qt8kg" pod "32c65765-23ce-437d-be07-4f136d09f263" (UID: "32c65765-23ce-437d-be07-4f136d09f263"). InnerVolumeSpecName "kube-api-access-qt8kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:36.873047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.873005    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt8kg\" (UniqueName: \"kubernetes.io/projected/32c65765-23ce-437d-be07-4f136d09f263-kube-api-access-qt8kg\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:36.873047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:36.873045    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c65765-23ce-437d-be07-4f136d09f263-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:37.371220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:37.371188    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:37.431629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:37.431590    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz" event={"ID":"32c65765-23ce-437d-be07-4f136d09f263","Type":"ContainerDied","Data":"f9a25cb9bc5d83ebfabccd7d7f16d81dd6ae36d67ddd746128151caeef31fa99"}
Oct 30 01:19:37.431844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:37.431658    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9a25cb9bc5d83ebfabccd7d7f16d81dd6ae36d67ddd746128151caeef31fa99"
Oct 30 01:19:37.431844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:37.431603    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"
Oct 30 01:19:38.181179 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:38.181146    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:38.630865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:38.630824    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:38.907333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:38.901943    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:39.181006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:39.180917    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:39.925660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:39.925623    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:40.537694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.537656    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"]
Oct 30 01:19:40.538601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.538579    4576 cpu_manager.go:414] "containerMap: removing container" podUID="32c65765-23ce-437d-be07-4f136d09f263" containerName="revoke-keys"
Oct 30 01:19:40.538661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.538604    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c65765-23ce-437d-be07-4f136d09f263" containerName="revoke-keys"
Oct 30 01:19:40.538799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.538786    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c65765-23ce-437d-be07-4f136d09f263" containerName="revoke-keys"
Oct 30 01:19:40.544433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.544408    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.552931 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.552897    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"]
Oct 30 01:19:40.558727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.558700    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.567837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.567807    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"]
Oct 30 01:19:40.594351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.594323    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"]
Oct 30 01:19:40.644571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.644523    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2fl\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-kube-api-access-dm2fl\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.644752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.644593    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.644752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.644721    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.644829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.644783    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.644869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.644847    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.644976 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.644956    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.645040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.645028    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.645085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.645058    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.645159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.645148    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8944c80f-2076-4692-aaea-d95d0f23d00e-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.746785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.746743    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.747127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747101    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.747339 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747307    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.747473 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747372    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.747473 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747453    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.747603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747539    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.747603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747553    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.747710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747599    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.747710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747636    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtjpl\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-kube-api-access-jtjpl\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.747710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747676    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6ef1e5-67c3-4889-bcd9-0e1260313283-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.747895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747745    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.747895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747758    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.747895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747805    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.747895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747873    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.748183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.747993    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.748183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.748022    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.748183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.748055    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.748183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.748094    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8944c80f-2076-4692-aaea-d95d0f23d00e-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.748183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.748136    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2fl\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-kube-api-access-dm2fl\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.748548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.748191    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.748548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.748275    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.749341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.749317    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8944c80f-2076-4692-aaea-d95d0f23d00e-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.749341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.749326    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.751453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.751428    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.751913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.751891    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.767684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.767629    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.772138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.772091    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2fl\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-kube-api-access-dm2fl\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.820324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.820216    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:40.849969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.849926    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.850197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.850011    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.850197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.850085    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.850197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.850121    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtjpl\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-kube-api-access-jtjpl\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.850197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.850158    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6ef1e5-67c3-4889-bcd9-0e1260313283-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.850516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.850197    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.850516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.850284    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.850516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.850316    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.850516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.850382    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.854708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.854615    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.854871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.854721    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-credential-socket\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.855194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.855170    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-data\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.855400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.855193    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-certs\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.855811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.855783    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6ef1e5-67c3-4889-bcd9-0e1260313283-istiod-ca-cert\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.862766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.862707    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-podinfo\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.863161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.863120    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-envoy\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.867947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.867918    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:40.880784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.880100    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-token\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.893151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.892925    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtjpl\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-kube-api-access-jtjpl\") pod \"e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") " pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:40.908518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:40.908489    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:41.213295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.213268    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"]
Oct 30 01:19:41.214599 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:41.214568    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8944c80f_2076_4692_aaea_d95d0f23d00e.slice/crio-f80400f043bccbf470d2cd8c7d9e928cb390ba1789e1a4f2aeeef2a5b3c3eb89 WatchSource:0}: Error finding container f80400f043bccbf470d2cd8c7d9e928cb390ba1789e1a4f2aeeef2a5b3c3eb89: Status 404 returned error can't find the container with id f80400f043bccbf470d2cd8c7d9e928cb390ba1789e1a4f2aeeef2a5b3c3eb89
Oct 30 01:19:41.217701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.217667    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:41.217831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.217765    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:41.217831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.217815    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:41.256606 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:19:41.256577    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6ef1e5_67c3_4889_bcd9_0e1260313283.slice/crio-10968b57089b154b1b5fd3a9baa91a417d973086f314860f43d1378ff746db42 WatchSource:0}: Error finding container 10968b57089b154b1b5fd3a9baa91a417d973086f314860f43d1378ff746db42: Status 404 returned error can't find the container with id 10968b57089b154b1b5fd3a9baa91a417d973086f314860f43d1378ff746db42
Oct 30 01:19:41.256827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.256805    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"]
Oct 30 01:19:41.260475 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.260434    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:41.260598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.260506    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:41.260598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.260539    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:19:41.468244 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.468136    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" event={"ID":"8944c80f-2076-4692-aaea-d95d0f23d00e","Type":"ContainerStarted","Data":"b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281"}
Oct 30 01:19:41.468244 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.468196    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" event={"ID":"8944c80f-2076-4692-aaea-d95d0f23d00e","Type":"ContainerStarted","Data":"f80400f043bccbf470d2cd8c7d9e928cb390ba1789e1a4f2aeeef2a5b3c3eb89"}
Oct 30 01:19:41.470854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.470822    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" event={"ID":"db6ef1e5-67c3-4889-bcd9-0e1260313283","Type":"ContainerStarted","Data":"b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f"}
Oct 30 01:19:41.470994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.470875    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" event={"ID":"db6ef1e5-67c3-4889-bcd9-0e1260313283","Type":"ContainerStarted","Data":"10968b57089b154b1b5fd3a9baa91a417d973086f314860f43d1378ff746db42"}
Oct 30 01:19:41.482277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.482250    4576 scope.go:122] "RemoveContainer" containerID="01b8c4ce399bca69e38bf73c16fc28d9e2335661c17a27a9b7c2f4d8d46df80d"
Oct 30 01:19:41.495443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.495392    4576 scope.go:122] "RemoveContainer" containerID="37fadb3db577e51d130cbef94f91da2c5fbb405f967560aeb535da3e429d351a"
Oct 30 01:19:41.511183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.510836    4576 scope.go:122] "RemoveContainer" containerID="a43c8aec59226858afeba41956d631ee2ed47cf09e69e170beee7f8d81a2aefa"
Oct 30 01:19:41.520904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.520846    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" podStartSLOduration=1.5208241789999999 podStartE2EDuration="1.520824179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:41.513560829 +0000 UTC m=+2461.584884666" watchObservedRunningTime="2026-10-30 01:19:41.520824179 +0000 UTC m=+2461.592148020"
Oct 30 01:19:41.527522 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.527397    4576 scope.go:122] "RemoveContainer" containerID="06444b2b780f0bca31f1543a3df8d28bb414435afbfd4966d4d28a5fe727396e"
Oct 30 01:19:41.544592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.544427    4576 scope.go:122] "RemoveContainer" containerID="cfc7ef71b733b52165afa5a8bbd0857efa390d68205b4bc0952fcc8a4c5f6605"
Oct 30 01:19:41.564810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.563083    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" podStartSLOduration=1.5630635750000001 podStartE2EDuration="1.563063575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:41.561559437 +0000 UTC m=+2461.632883274" watchObservedRunningTime="2026-10-30 01:19:41.563063575 +0000 UTC m=+2461.634387415"
Oct 30 01:19:41.868830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.868753    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:41.909278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:41.909248    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:42.139464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.134973    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"]
Oct 30 01:19:42.156649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.154861    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"]
Oct 30 01:19:42.156649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.155033    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.160709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.160681    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:42.227274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.211911    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-24d5ce\"/\"default-dockercfg-sscjg\""
Oct 30 01:19:42.227274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.212105    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-24d5ce\"/\"rate-test-model-24d5ce-kserve-self-signed-certs\""
Oct 30 01:19:42.227274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.212209    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-24d5ce\"/\"kube-root-ca.crt\""
Oct 30 01:19:42.227274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.225534    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-24d5ce\"/\"openshift-service-ca.crt\""
Oct 30 01:19:42.316259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.315281    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-model-cache\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.316259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.315348    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tmp-dir\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.316259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.315388    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcflw\" (UniqueName: \"kubernetes.io/projected/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-kube-api-access-kcflw\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.316259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.315431    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-home\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.316259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.315524    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-dshm\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.316259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.315714    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tls-certs\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.422361    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tls-certs\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.422502    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-model-cache\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.422547    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tmp-dir\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.422582    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcflw\" (UniqueName: \"kubernetes.io/projected/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-kube-api-access-kcflw\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.422622    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-home\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.422707    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-dshm\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.423609    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tmp-dir\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.424199    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-model-cache\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.424923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.424571    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-home\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.485605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.445608    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-dshm\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.485605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.451399    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tls-certs\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.485605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.466035    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcflw\" (UniqueName: \"kubernetes.io/projected/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-kube-api-access-kcflw\") pod \"rate-test-model-24d5ce-kserve-565ff54df7-khdt5\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") " pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.564882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.564848    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:42.917421 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.917372    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:19:42.935012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.933434    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:19:42.935012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.934885    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:19:42.935012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.934925    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:19:42.963341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:42.963288    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:43.062153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.062072    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"]
Oct 30 01:19:43.147022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.146827    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:43.257086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.256810    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:43.522260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.521770    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:43.539741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.539706    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" event={"ID":"5f2f69c0-1ec3-460d-b8a4-c79f166a9870","Type":"ContainerStarted","Data":"90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0"}
Oct 30 01:19:43.539975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.539960    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" event={"ID":"5f2f69c0-1ec3-460d-b8a4-c79f166a9870","Type":"ContainerStarted","Data":"2337964c3fd0bbe8f3d3014412f3db7dbeb1ae8a883221a9b4f12ef66b1bf5ae"}
Oct 30 01:19:43.567102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.567053    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:43.598259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.597758    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" podStartSLOduration=1.597736826 podStartE2EDuration="1.597736826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:43.585804446 +0000 UTC m=+2463.657128315" watchObservedRunningTime="2026-10-30 01:19:43.597736826 +0000 UTC m=+2463.669060664"
Oct 30 01:19:43.877313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.871760    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:19:43.877313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.871826    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:19:43.913258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.912646    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:19:43.913258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:43.912709    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:19:44.234849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.234811    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"]
Oct 30 01:19:44.239688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.239665    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.256637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.256611    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"]
Oct 30 01:19:44.265604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.265578    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:19:44.266052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.265996    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:19:44.275257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.272172    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.275257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.272297    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftggm\" (UniqueName: \"kubernetes.io/projected/514c40e0-62b3-4df2-8bf2-428d5dce63a3-kube-api-access-ftggm\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.275257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.272431    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.299426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.299396    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:44.397966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.397927    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.398496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.398466    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.398783 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.398761    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftggm\" (UniqueName: \"kubernetes.io/projected/514c40e0-62b3-4df2-8bf2-428d5dce63a3-kube-api-access-ftggm\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.420260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.400113    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.441900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.440968    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:19:44.442965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.442924    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftggm\" (UniqueName: \"kubernetes.io/projected/514c40e0-62b3-4df2-8bf2-428d5dce63a3-kube-api-access-ftggm\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.443169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.443145    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:19:44.444056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.444032    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7902333c-54zr6\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.555070 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.554965    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:44.890318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.881767    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:19:44.890318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.881827    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:19:44.917288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.915749    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:19:44.917288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:44.915814    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:19:45.114617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.113640    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:45.244704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.244670    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:45.333830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.333797    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"]
Oct 30 01:19:45.429297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.419202    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:45.678177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.637716    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6" event={"ID":"514c40e0-62b3-4df2-8bf2-428d5dce63a3","Type":"ContainerStarted","Data":"ffd6bcb0ae76ede1ae3fd5f6f6625e7f8d2e276c5ce407837ef03195e8defc22"}
Oct 30 01:19:45.872146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.872112    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:19:45.872850 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.872801    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:19:45.915345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.915310    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:19:45.915675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:45.915629    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:19:46.868871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:46.868833    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:19:46.970316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:46.955189    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:19:46.970316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:46.910206    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:19:46.970316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:46.955743    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:19:46.989758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:46.989718    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6" event={"ID":"514c40e0-62b3-4df2-8bf2-428d5dce63a3","Type":"ContainerStarted","Data":"dda05d813602d8b8a8bd48e2893f9d5e5e7a44d80a95a81858c0568d9d938540"}
Oct 30 01:19:47.114293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.113997    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs" podUID="3d5ec9dc-591c-43da-a9fa-ed124cebc38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.177:8000/ready\": context deadline exceeded"
Oct 30 01:19:47.121269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.114984    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:47.127178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.127137    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:47.337321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.333344    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:47.576702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.576650    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:47.577697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.577663    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:47.822256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.822191    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:47.869696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.869659    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:19:47.870373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.870340    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:19:47.914002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.913938    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:47.919638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.919604    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:19:47.919893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:47.919863    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:19:48.018657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.018606    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:48.044356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.042477    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:19:48.057645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.057088    4576 generic.go:356] "Generic (PLEG): container finished" podID="514c40e0-62b3-4df2-8bf2-428d5dce63a3" containerID="dda05d813602d8b8a8bd48e2893f9d5e5e7a44d80a95a81858c0568d9d938540" exitCode=0
Oct 30 01:19:48.057645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.057328    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6" event={"ID":"514c40e0-62b3-4df2-8bf2-428d5dce63a3","Type":"ContainerDied","Data":"dda05d813602d8b8a8bd48e2893f9d5e5e7a44d80a95a81858c0568d9d938540"}
Oct 30 01:19:48.425621 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:19:48.425583    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:19:48.873470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.873438    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:48.873924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.873825    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:48.874658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.874635    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:19:48.926649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.926619    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:48.927023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.927003    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:48.928141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:48.928119    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:19:49.277841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.277816    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:49.318695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.318666    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-xks-ca\") pod \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") "
Oct 30 01:19:49.318856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.318718    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514c40e0-62b3-4df2-8bf2-428d5dce63a3-kube-api-access-ftggm\" (UniqueName: \"kubernetes.io/projected/514c40e0-62b3-4df2-8bf2-428d5dce63a3-kube-api-access-ftggm\") pod \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") "
Oct 30 01:19:49.318856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.318770    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-openshift-ca\") pod \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\" (UID: \"514c40e0-62b3-4df2-8bf2-428d5dce63a3\") "
Oct 30 01:19:49.319203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.319169    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-openshift-ca" pod "514c40e0-62b3-4df2-8bf2-428d5dce63a3" (UID: "514c40e0-62b3-4df2-8bf2-428d5dce63a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:49.319515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.319493    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:49.328618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.324159    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514c40e0-62b3-4df2-8bf2-428d5dce63a3-kube-api-access-ftggm" pod "514c40e0-62b3-4df2-8bf2-428d5dce63a3" (UID: "514c40e0-62b3-4df2-8bf2-428d5dce63a3"). InnerVolumeSpecName "kube-api-access-ftggm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:49.338699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.338651    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-xks-ca" pod "514c40e0-62b3-4df2-8bf2-428d5dce63a3" (UID: "514c40e0-62b3-4df2-8bf2-428d5dce63a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:49.421446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.421354    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514c40e0-62b3-4df2-8bf2-428d5dce63a3-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:49.421446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.421380    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftggm\" (UniqueName: \"kubernetes.io/projected/514c40e0-62b3-4df2-8bf2-428d5dce63a3-kube-api-access-ftggm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:19:49.583630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:49.583602    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:50.074326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:50.074296    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"
Oct 30 01:19:50.074737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:50.074301    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6" event={"ID":"514c40e0-62b3-4df2-8bf2-428d5dce63a3","Type":"ContainerDied","Data":"ffd6bcb0ae76ede1ae3fd5f6f6625e7f8d2e276c5ce407837ef03195e8defc22"}
Oct 30 01:19:50.074737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:50.074424    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd6bcb0ae76ede1ae3fd5f6f6625e7f8d2e276c5ce407837ef03195e8defc22"
Oct 30 01:19:51.087764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:51.087724    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:52.565708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:52.565666    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:52.579274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:52.579227    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:53.110199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:53.110172    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:19:53.173892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:53.173857    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:54.604633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:54.604597    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:54.712631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:54.712594    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:55.096476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:55.096440    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:56.454090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:56.454052    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:58.030439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:58.030402    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"]
Oct 30 01:19:58.043866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:58.043837    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9e53ae-7b84t"]
Oct 30 01:19:58.114741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:58.114706    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0780e8f4-f5c6-4511-bc88-5166f857df8d" path="/var/lib/kubelet/pods/0780e8f4-f5c6-4511-bc88-5166f857df8d/volumes"
Oct 30 01:19:58.260186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:58.260149    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:19:59.898121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:19:59.898059    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:01.948388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:01.948359    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:03.558678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:03.558639    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:05.056950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:05.056915    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:06.566960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:06.566922    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:08.172435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:08.172400    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:09.029372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:09.029338    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"]
Oct 30 01:20:09.043176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:09.043145    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b97a911-lsklb"]
Oct 30 01:20:09.623385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:09.623352    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:10.111821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:10.111791    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2" path="/var/lib/kubelet/pods/e6e86a82-5c14-4287-a9d9-05c7b4e7d1b2/volumes"
Oct 30 01:20:11.082593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:11.082559    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:12.619769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:12.619731    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"]
Oct 30 01:20:12.676565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:12.676402    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:12.808316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:12.808274    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:12.941348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:12.939852    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:13.107705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.107666    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"]
Oct 30 01:20:13.108662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.108645    4576 cpu_manager.go:414] "containerMap: removing container" podUID="514c40e0-62b3-4df2-8bf2-428d5dce63a3" containerName="revoke-keys"
Oct 30 01:20:13.108724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.108665    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514c40e0-62b3-4df2-8bf2-428d5dce63a3" containerName="revoke-keys"
Oct 30 01:20:13.108809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.108798    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="514c40e0-62b3-4df2-8bf2-428d5dce63a3" containerName="revoke-keys"
Oct 30 01:20:13.113328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.113307    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.129566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.129544    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:13.129742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.129630    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:20:13.130759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.130739    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"]
Oct 30 01:20:13.170277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.170223    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.171006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.170962    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.171411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.171387    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvb6\" (UniqueName: \"kubernetes.io/projected/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-kube-api-access-vpvb6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.272557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.272482    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvb6\" (UniqueName: \"kubernetes.io/projected/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-kube-api-access-vpvb6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.272722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.272632    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.272787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.272774    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.273309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.273285    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.275273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.275251    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.293030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.293006    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvb6\" (UniqueName: \"kubernetes.io/projected/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-kube-api-access-vpvb6\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.433069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.433033    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:13.746059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:13.746029    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"]
Oct 30 01:20:13.746506 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:20:13.746484    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac26ca1c_3ab2_49ec_a5ec_8b3f03cefdb7.slice/crio-50e82f416e51a421c977abcbc51f1b0023ac10df316f7b89ae105dd1c2ca6137 WatchSource:0}: Error finding container 50e82f416e51a421c977abcbc51f1b0023ac10df316f7b89ae105dd1c2ca6137: Status 404 returned error can't find the container with id 50e82f416e51a421c977abcbc51f1b0023ac10df316f7b89ae105dd1c2ca6137
Oct 30 01:20:14.307122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:14.307013    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj" event={"ID":"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7","Type":"ContainerStarted","Data":"fb48dcbd99402de8ff25a8a9c27f33030d9c3e0f033c6f603a1b9e8cf15726d6"}
Oct 30 01:20:14.307122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:14.307094    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj" event={"ID":"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7","Type":"ContainerStarted","Data":"50e82f416e51a421c977abcbc51f1b0023ac10df316f7b89ae105dd1c2ca6137"}
Oct 30 01:20:14.349810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:14.349755    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj" podStartSLOduration=1.349740438 podStartE2EDuration="1.349740438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:14.348598232 +0000 UTC m=+2494.419922087" watchObservedRunningTime="2026-10-30 01:20:14.349740438 +0000 UTC m=+2494.421064272"
Oct 30 01:20:14.768647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:14.768610    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:15.316261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:15.316216    4576 generic.go:356] "Generic (PLEG): container finished" podID="ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7" containerID="fb48dcbd99402de8ff25a8a9c27f33030d9c3e0f033c6f603a1b9e8cf15726d6" exitCode=0
Oct 30 01:20:15.316261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:15.316266    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj" event={"ID":"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7","Type":"ContainerDied","Data":"fb48dcbd99402de8ff25a8a9c27f33030d9c3e0f033c6f603a1b9e8cf15726d6"}
Oct 30 01:20:16.534957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.534921    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:16.662566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.662504    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:16.853920 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.853890    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-kube-api-access-vpvb6\" (UniqueName: \"kubernetes.io/projected/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-kube-api-access-vpvb6\") pod \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") "
Oct 30 01:20:16.854087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.854076    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-openshift-ca\") pod \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") "
Oct 30 01:20:16.854138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.854114    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-xks-ca\") pod \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\" (UID: \"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7\") "
Oct 30 01:20:16.854417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.854387    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-openshift-ca" pod "ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7" (UID: "ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:16.854674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.854659    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:16.856219 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.856199    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-kube-api-access-vpvb6" pod "ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7" (UID: "ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7"). InnerVolumeSpecName "kube-api-access-vpvb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:16.856337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.856316    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-xks-ca" pod "ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7" (UID: "ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:16.956132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.956097    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:16.956132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:16.956134    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpvb6\" (UniqueName: \"kubernetes.io/projected/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7-kube-api-access-vpvb6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:17.429463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:17.429373    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"
Oct 30 01:20:17.429463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:17.429385    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj" event={"ID":"ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7","Type":"ContainerDied","Data":"50e82f416e51a421c977abcbc51f1b0023ac10df316f7b89ae105dd1c2ca6137"}
Oct 30 01:20:17.429655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:17.429475    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50e82f416e51a421c977abcbc51f1b0023ac10df316f7b89ae105dd1c2ca6137"
Oct 30 01:20:17.497957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:17.497894    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:17.775027 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:17.774990    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"]
Oct 30 01:20:17.920282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:17.920197    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:18.032547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.031917    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:18.208535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.206551    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:18.414128 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:20:18.414028    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:20:18.906276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.906244    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"]
Oct 30 01:20:18.906913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.906897    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7" containerName="revoke-keys"
Oct 30 01:20:18.906969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.906914    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7" containerName="revoke-keys"
Oct 30 01:20:18.907036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.907027    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7" containerName="revoke-keys"
Oct 30 01:20:18.911121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.911096    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:18.939054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.939018    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"]
Oct 30 01:20:18.940635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.940584    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-cfe8f0\"/\"openshift-service-ca.crt\""
Oct 30 01:20:18.940635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.940588    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-cfe8f0\"/\"kube-root-ca.crt\""
Oct 30 01:20:18.941732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.941708    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-model-cache\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:18.941860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.941774    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-dshm\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:18.941906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.941882    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-home\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:18.942007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.941987    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btrgp\" (UniqueName: \"kubernetes.io/projected/10857aa4-9e02-4728-8581-c32e865769a6-kube-api-access-btrgp\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:18.942105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.942090    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10857aa4-9e02-4728-8581-c32e865769a6-tls-certs\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:18.942246 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.942217    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-tmp-dir\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.000302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:18.997132    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:19.011758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.011730    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-cfe8f0\"/\"rate-test-model-cfe8f0-kserve-self-signed-certs\""
Oct 30 01:20:19.011933 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.011815    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-cfe8f0\"/\"default-dockercfg-mfnh4\""
Oct 30 01:20:19.043605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.043573    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-model-cache\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.043769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.043631    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-dshm\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.043769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.043712    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-home\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.043845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.043780    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btrgp\" (UniqueName: \"kubernetes.io/projected/10857aa4-9e02-4728-8581-c32e865769a6-kube-api-access-btrgp\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.043905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.043890    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10857aa4-9e02-4728-8581-c32e865769a6-tls-certs\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.044027 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.044005    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-model-cache\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.044092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.044033    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-tmp-dir\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.044157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.044140    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-home\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.044400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.044337    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-tmp-dir\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.046203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.046179    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-dshm\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.046735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.046713    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10857aa4-9e02-4728-8581-c32e865769a6-tls-certs\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.064671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.064641    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btrgp\" (UniqueName: \"kubernetes.io/projected/10857aa4-9e02-4728-8581-c32e865769a6-kube-api-access-btrgp\") pod \"rate-test-model-cfe8f0-kserve-5ff748b677-f75fr\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") " pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.240256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.239484    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:19.696413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.696380    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"]
Oct 30 01:20:19.698055 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:20:19.698020    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10857aa4_9e02_4728_8581_c32e865769a6.slice/crio-674b3957fd705791691ee34643cf0e48c72a21d2dfa8386b096f4dda2e5f6f6e WatchSource:0}: Error finding container 674b3957fd705791691ee34643cf0e48c72a21d2dfa8386b096f4dda2e5f6f6e: Status 404 returned error can't find the container with id 674b3957fd705791691ee34643cf0e48c72a21d2dfa8386b096f4dda2e5f6f6e
Oct 30 01:20:19.863419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:19.863386    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:20.511635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:20.511596    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" event={"ID":"10857aa4-9e02-4728-8581-c32e865769a6","Type":"ContainerStarted","Data":"6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886"}
Oct 30 01:20:20.511635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:20.511635    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" event={"ID":"10857aa4-9e02-4728-8581-c32e865769a6","Type":"ContainerStarted","Data":"674b3957fd705791691ee34643cf0e48c72a21d2dfa8386b096f4dda2e5f6f6e"}
Oct 30 01:20:20.555004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:20.554952    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podStartSLOduration=2.554937106 podStartE2EDuration="2.554937106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:20.552859971 +0000 UTC m=+2500.624183808" watchObservedRunningTime="2026-10-30 01:20:20.554937106 +0000 UTC m=+2500.626260940"
Oct 30 01:20:21.169579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:21.158213    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:20:21.241378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:21.241341    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:21.793259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:21.792754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:22.794267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:22.787306    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb" podUID="83d26335-e3da-4e79-ac32-1ca6851556ee" containerName="payload-processing" containerID="cri-o://af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394" gracePeriod=25
Oct 30 01:20:23.318823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.294764    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:20:23.441213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.441185    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d26335-e3da-4e79-ac32-1ca6851556ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d26335-e3da-4e79-ac32-1ca6851556ee-plugins-config-volume\") pod \"83d26335-e3da-4e79-ac32-1ca6851556ee\" (UID: \"83d26335-e3da-4e79-ac32-1ca6851556ee\") "
Oct 30 01:20:23.441453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.441262    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d26335-e3da-4e79-ac32-1ca6851556ee-kube-api-access-w7wz5\" (UniqueName: \"kubernetes.io/projected/83d26335-e3da-4e79-ac32-1ca6851556ee-kube-api-access-w7wz5\") pod \"83d26335-e3da-4e79-ac32-1ca6851556ee\" (UID: \"83d26335-e3da-4e79-ac32-1ca6851556ee\") "
Oct 30 01:20:23.441670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.441643    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d26335-e3da-4e79-ac32-1ca6851556ee-plugins-config-volume" pod "83d26335-e3da-4e79-ac32-1ca6851556ee" (UID: "83d26335-e3da-4e79-ac32-1ca6851556ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:23.441853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.441836    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d26335-e3da-4e79-ac32-1ca6851556ee-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:23.443640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.443603    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d26335-e3da-4e79-ac32-1ca6851556ee-kube-api-access-w7wz5" pod "83d26335-e3da-4e79-ac32-1ca6851556ee" (UID: "83d26335-e3da-4e79-ac32-1ca6851556ee"). InnerVolumeSpecName "kube-api-access-w7wz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:23.534842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.534811    4576 generic.go:356] "Generic (PLEG): container finished" podID="83d26335-e3da-4e79-ac32-1ca6851556ee" containerID="af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394" exitCode=0
Oct 30 01:20:23.535016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.534865    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"
Oct 30 01:20:23.535016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.534880    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb" event={"ID":"83d26335-e3da-4e79-ac32-1ca6851556ee","Type":"ContainerDied","Data":"af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394"}
Oct 30 01:20:23.535016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.534939    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb" event={"ID":"83d26335-e3da-4e79-ac32-1ca6851556ee","Type":"ContainerDied","Data":"bd39284e7063c20edde6a671e4f158ab954f826a7daf5afd5d2e3450838e17d1"}
Oct 30 01:20:23.535016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.534985    4576 scope.go:122] "RemoveContainer" containerID="af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394"
Oct 30 01:20:23.543192 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.543159    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wz5\" (UniqueName: \"kubernetes.io/projected/83d26335-e3da-4e79-ac32-1ca6851556ee-kube-api-access-w7wz5\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:23.548733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.548704    4576 scope.go:122] "RemoveContainer" containerID="af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394"
Oct 30 01:20:23.549122 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:20:23.549102    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394\": container with ID starting with af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394 not found: ID does not exist" containerID="af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394"
Oct 30 01:20:23.549193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.549130    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394"} err="failed to get container status \"af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394\": rpc error: code = NotFound desc = could not find container \"af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394\": container with ID starting with af6cc8a0b8653529c0979b9c22f9245f8571eecf339d9f766e6aa5413fbbb394 not found: ID does not exist"
Oct 30 01:20:23.601225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.601185    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"]
Oct 30 01:20:23.605581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.605548    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5c515e-65888c89f6-pqflb"]
Oct 30 01:20:23.622808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.622749    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:23.910965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:23.910287    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:24.042014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:24.041972    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:24.122895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:24.122862    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d26335-e3da-4e79-ac32-1ca6851556ee" path="/var/lib/kubelet/pods/83d26335-e3da-4e79-ac32-1ca6851556ee/volumes"
Oct 30 01:20:25.309303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.305846    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"]
Oct 30 01:20:25.309303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.307094    4576 cpu_manager.go:414] "containerMap: removing container" podUID="83d26335-e3da-4e79-ac32-1ca6851556ee" containerName="payload-processing"
Oct 30 01:20:25.309303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.307117    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d26335-e3da-4e79-ac32-1ca6851556ee" containerName="payload-processing"
Oct 30 01:20:25.309303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.307368    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d26335-e3da-4e79-ac32-1ca6851556ee" containerName="payload-processing"
Oct 30 01:20:25.314953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.313876    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.329900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.329867    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"]
Oct 30 01:20:25.330369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.330344    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:25.336629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.336600    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:20:25.363508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.360914    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:25.418371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.418336    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.424967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.421587    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnz86\" (UniqueName: \"kubernetes.io/projected/15aecdd0-cb38-43c1-94f9-dae28674ff13-kube-api-access-vnz86\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.424967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.421747    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.525427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.525366    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.525655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.525464    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnz86\" (UniqueName: \"kubernetes.io/projected/15aecdd0-cb38-43c1-94f9-dae28674ff13-kube-api-access-vnz86\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.525655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.525520    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.526167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.526095    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.528783 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.528756    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.547220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.547187    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnz86\" (UniqueName: \"kubernetes.io/projected/15aecdd0-cb38-43c1-94f9-dae28674ff13-kube-api-access-vnz86\") pod \"maas-api-revoke-sub-8df5ef4a-sg8nk\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:25.630829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:25.630741    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:26.002403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:26.002338    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"]
Oct 30 01:20:26.002548 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:20:26.002478    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15aecdd0_cb38_43c1_94f9_dae28674ff13.slice/crio-a715be0451e62b5e64d4c0d9f7e53a2879a5cbb11398e710d6888319edbee235 WatchSource:0}: Error finding container a715be0451e62b5e64d4c0d9f7e53a2879a5cbb11398e710d6888319edbee235: Status 404 returned error can't find the container with id a715be0451e62b5e64d4c0d9f7e53a2879a5cbb11398e710d6888319edbee235
Oct 30 01:20:26.611406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:26.611373    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk" event={"ID":"15aecdd0-cb38-43c1-94f9-dae28674ff13","Type":"ContainerStarted","Data":"25584fbee8dc641dbc9d856b9afd8fbe47dafdec62a8f247975646a21b87dad2"}
Oct 30 01:20:26.611875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:26.611429    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk" event={"ID":"15aecdd0-cb38-43c1-94f9-dae28674ff13","Type":"ContainerStarted","Data":"a715be0451e62b5e64d4c0d9f7e53a2879a5cbb11398e710d6888319edbee235"}
Oct 30 01:20:26.655124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:26.655072    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk" podStartSLOduration=1.655058916 podStartE2EDuration="1.655058916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:26.653431615 +0000 UTC m=+2506.724755453" watchObservedRunningTime="2026-10-30 01:20:26.655058916 +0000 UTC m=+2506.726382751"
Oct 30 01:20:26.894006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:26.893910    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:27.620375 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:27.620294    4576 generic.go:356] "Generic (PLEG): container finished" podID="15aecdd0-cb38-43c1-94f9-dae28674ff13" containerID="25584fbee8dc641dbc9d856b9afd8fbe47dafdec62a8f247975646a21b87dad2" exitCode=0
Oct 30 01:20:27.620758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:27.620367    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk" event={"ID":"15aecdd0-cb38-43c1-94f9-dae28674ff13","Type":"ContainerDied","Data":"25584fbee8dc641dbc9d856b9afd8fbe47dafdec62a8f247975646a21b87dad2"}
Oct 30 01:20:27.698463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:27.696407    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs" podUID="3d5ec9dc-591c-43da-a9fa-ed124cebc38e" containerName="main" containerID="cri-o://2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612" gracePeriod=45
Oct 30 01:20:28.029891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.029857    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:20:28.066551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.066520    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-dshm\") pod \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") "
Oct 30 01:20:28.066742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.066605    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-home\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-home\") pod \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") "
Oct 30 01:20:28.066813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.066747    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-kube-api-access-98ctf\" (UniqueName: \"kubernetes.io/projected/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-kube-api-access-98ctf\") pod \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") "
Oct 30 01:20:28.066934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.066906    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-home" pod "3d5ec9dc-591c-43da-a9fa-ed124cebc38e" (UID: "3d5ec9dc-591c-43da-a9fa-ed124cebc38e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:28.067008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.066917    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-model-cache\") pod \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") "
Oct 30 01:20:28.067066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.067038    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tls-certs\") pod \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") "
Oct 30 01:20:28.067151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.067131    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-model-cache" pod "3d5ec9dc-591c-43da-a9fa-ed124cebc38e" (UID: "3d5ec9dc-591c-43da-a9fa-ed124cebc38e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:28.067224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.067134    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tmp-dir\") pod \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\" (UID: \"3d5ec9dc-591c-43da-a9fa-ed124cebc38e\") "
Oct 30 01:20:28.089468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.080558    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.089468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.080594    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.094256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.090339    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tmp-dir" pod "3d5ec9dc-591c-43da-a9fa-ed124cebc38e" (UID: "3d5ec9dc-591c-43da-a9fa-ed124cebc38e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:28.102193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.101607    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-kube-api-access-98ctf" pod "3d5ec9dc-591c-43da-a9fa-ed124cebc38e" (UID: "3d5ec9dc-591c-43da-a9fa-ed124cebc38e"). InnerVolumeSpecName "kube-api-access-98ctf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:28.102729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.102700    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-dshm" pod "3d5ec9dc-591c-43da-a9fa-ed124cebc38e" (UID: "3d5ec9dc-591c-43da-a9fa-ed124cebc38e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:28.122374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.122336    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tls-certs" pod "3d5ec9dc-591c-43da-a9fa-ed124cebc38e" (UID: "3d5ec9dc-591c-43da-a9fa-ed124cebc38e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:28.193320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.193276    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ctf\" (UniqueName: \"kubernetes.io/projected/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-kube-api-access-98ctf\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.193320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.193311    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.193547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.193327    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.193547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.193338    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5ec9dc-591c-43da-a9fa-ed124cebc38e-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:28.320444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.320414    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"]
Oct 30 01:20:28.336658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.321079    4576 cpu_manager.go:414] "containerMap: removing container" podUID="3d5ec9dc-591c-43da-a9fa-ed124cebc38e" containerName="main"
Oct 30 01:20:28.336658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.321094    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5ec9dc-591c-43da-a9fa-ed124cebc38e" containerName="main"
Oct 30 01:20:28.336658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.321251    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5ec9dc-591c-43da-a9fa-ed124cebc38e" containerName="main"
Oct 30 01:20:28.336658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.324858    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.344392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.341497    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5c5a66-serving-cert\""
Oct 30 01:20:28.352298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.346814    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"]
Oct 30 01:20:28.408493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.408141    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnz2\" (UniqueName: \"kubernetes.io/projected/c592358a-3ff2-478f-a681-2a7c50d79afa-kube-api-access-gsnz2\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.408493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.408198    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.408493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.408325    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.408493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.408404    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c592358a-3ff2-478f-a681-2a7c50d79afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.437158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.437117    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:28.527195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.512287    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnz2\" (UniqueName: \"kubernetes.io/projected/c592358a-3ff2-478f-a681-2a7c50d79afa-kube-api-access-gsnz2\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.527195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.512334    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.527195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.512432    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.527195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.512496    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c592358a-3ff2-478f-a681-2a7c50d79afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.527195 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:20:28.520360    4576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-serving-cert: secret "maas-api-e2e-ipp-b-5c5a66-serving-cert" not found
Oct 30 01:20:28.527195 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:20:28.521875    4576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls podName:c592358a-3ff2-478f-a681-2a7c50d79afa nodeName:}" failed. No retries permitted until 2026-10-30 01:20:29.021848984 +0000 UTC m=+2509.093172800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls") pod "maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz" (UID: "c592358a-3ff2-478f-a681-2a7c50d79afa") : secret "maas-api-e2e-ipp-b-5c5a66-serving-cert" not found
Oct 30 01:20:28.527195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.521757    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c592358a-3ff2-478f-a681-2a7c50d79afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.537502 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.530339    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.555730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.555699    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnz2\" (UniqueName: \"kubernetes.io/projected/c592358a-3ff2-478f-a681-2a7c50d79afa-kube-api-access-gsnz2\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:28.630609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.630531    4576 generic.go:356] "Generic (PLEG): container finished" podID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerID="0d372a03d8c6df1584fb849e279e326bdad9739794af16b76d2478d28302d10e" exitCode=0
Oct 30 01:20:28.631014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.630601    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" event={"ID":"81e867f5-4e74-42b1-941b-8d5e8e6b56d5","Type":"ContainerDied","Data":"0d372a03d8c6df1584fb849e279e326bdad9739794af16b76d2478d28302d10e"}
Oct 30 01:20:28.631173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.631155    4576 scope.go:122] "RemoveContainer" containerID="0d372a03d8c6df1584fb849e279e326bdad9739794af16b76d2478d28302d10e"
Oct 30 01:20:28.632420 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.632396    4576 generic.go:356] "Generic (PLEG): container finished" podID="3d5ec9dc-591c-43da-a9fa-ed124cebc38e" containerID="2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612" exitCode=0
Oct 30 01:20:28.632545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.632463    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"
Oct 30 01:20:28.632545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.632522    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs" event={"ID":"3d5ec9dc-591c-43da-a9fa-ed124cebc38e","Type":"ContainerDied","Data":"2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612"}
Oct 30 01:20:28.632545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.632543    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs" event={"ID":"3d5ec9dc-591c-43da-a9fa-ed124cebc38e","Type":"ContainerDied","Data":"ffb510d450f38191ab0f20a433accbd6b3d734476b4cfeea89df0d5930663348"}
Oct 30 01:20:28.632667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.632557    4576 scope.go:122] "RemoveContainer" containerID="2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612"
Oct 30 01:20:28.633476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.633441    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:28.633585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.633533    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:28.633677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.633581    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:28.652437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.651833    4576 scope.go:122] "RemoveContainer" containerID="2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612"
Oct 30 01:20:28.652437 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:20:28.652117    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612\": container with ID starting with 2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612 not found: ID does not exist" containerID="2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612"
Oct 30 01:20:28.652437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.652142    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612"} err="failed to get container status \"2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612\": rpc error: code = NotFound desc = could not find container \"2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612\": container with ID starting with 2b6210416fc5807d23081af891c115a04fe947ffd668c0c989a4e4f4e2e66612 not found: ID does not exist"
Oct 30 01:20:28.655171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.655139    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:28.748988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.741356    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"]
Oct 30 01:20:28.749377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.749348    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43d152/sub-test-model-43d152-kserve-595f57d84c-jnxfs"]
Oct 30 01:20:28.762802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.762776    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:28.825637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.825606    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:20:28.919324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:28.918565    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:29.038919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.038820    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-openshift-ca\") pod \"15aecdd0-cb38-43c1-94f9-dae28674ff13\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") "
Oct 30 01:20:29.038919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.038881    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-xks-ca\") pod \"15aecdd0-cb38-43c1-94f9-dae28674ff13\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") "
Oct 30 01:20:29.039156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.038990    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15aecdd0-cb38-43c1-94f9-dae28674ff13-kube-api-access-vnz86\" (UniqueName: \"kubernetes.io/projected/15aecdd0-cb38-43c1-94f9-dae28674ff13-kube-api-access-vnz86\") pod \"15aecdd0-cb38-43c1-94f9-dae28674ff13\" (UID: \"15aecdd0-cb38-43c1-94f9-dae28674ff13\") "
Oct 30 01:20:29.039282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.039215    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-openshift-ca" pod "15aecdd0-cb38-43c1-94f9-dae28674ff13" (UID: "15aecdd0-cb38-43c1-94f9-dae28674ff13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:29.039602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.039577    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:29.039757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.039743    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:29.041933 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.041902    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-xks-ca" pod "15aecdd0-cb38-43c1-94f9-dae28674ff13" (UID: "15aecdd0-cb38-43c1-94f9-dae28674ff13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:29.042072 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.042032    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15aecdd0-cb38-43c1-94f9-dae28674ff13-kube-api-access-vnz86" pod "15aecdd0-cb38-43c1-94f9-dae28674ff13" (UID: "15aecdd0-cb38-43c1-94f9-dae28674ff13"). InnerVolumeSpecName "kube-api-access-vnz86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:29.042156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.042076    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:29.141349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.141313    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15aecdd0-cb38-43c1-94f9-dae28674ff13-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:29.141349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.141343    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnz86\" (UniqueName: \"kubernetes.io/projected/15aecdd0-cb38-43c1-94f9-dae28674ff13-kube-api-access-vnz86\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:29.240660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.240481    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:29.244180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.244149    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:29.264947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.264608    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:29.621594 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:20:29.621534    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc592358a_3ff2_478f_a681_2a7c50d79afa.slice/crio-262efdff133a1b2638f59a0b0965edf80d56e3c009571cedbda66235d5f8cca5 WatchSource:0}: Error finding container 262efdff133a1b2638f59a0b0965edf80d56e3c009571cedbda66235d5f8cca5: Status 404 returned error can't find the container with id 262efdff133a1b2638f59a0b0965edf80d56e3c009571cedbda66235d5f8cca5
Oct 30 01:20:29.624147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.624121    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"]
Oct 30 01:20:29.643489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.643449    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" event={"ID":"81e867f5-4e74-42b1-941b-8d5e8e6b56d5","Type":"ContainerStarted","Data":"f3f3d5d6d847edf4235e651b796b4f0e72a8d17dc355ede9d708d3332c13dd1d"}
Oct 30 01:20:29.645198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.645169    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz" event={"ID":"c592358a-3ff2-478f-a681-2a7c50d79afa","Type":"ContainerStarted","Data":"262efdff133a1b2638f59a0b0965edf80d56e3c009571cedbda66235d5f8cca5"}
Oct 30 01:20:29.647856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.647833    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"
Oct 30 01:20:29.647963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.647856    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk" event={"ID":"15aecdd0-cb38-43c1-94f9-dae28674ff13","Type":"ContainerDied","Data":"a715be0451e62b5e64d4c0d9f7e53a2879a5cbb11398e710d6888319edbee235"}
Oct 30 01:20:29.647963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.647897    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a715be0451e62b5e64d4c0d9f7e53a2879a5cbb11398e710d6888319edbee235"
Oct 30 01:20:29.667196 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:29.667173    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:20:30.114435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:30.114401    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5ec9dc-591c-43da-a9fa-ed124cebc38e" path="/var/lib/kubelet/pods/3d5ec9dc-591c-43da-a9fa-ed124cebc38e/volumes"
Oct 30 01:20:30.192623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:30.192587    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:20:30.197180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:30.196841    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:20:30.197180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:30.196907    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 01:20:30.662609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:30.662572    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz" event={"ID":"c592358a-3ff2-478f-a681-2a7c50d79afa","Type":"ContainerStarted","Data":"ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552"}
Oct 30 01:20:30.663107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:30.662953    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:30.711059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:30.711006    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz" podStartSLOduration=2.21335717 podStartE2EDuration="2.710989609s" totalImagesPullingTime="497.632439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:30.709326159 +0000 UTC m=+2510.780649994" watchObservedRunningTime="2026-10-30 01:20:30.710989609 +0000 UTC m=+2510.782313444"
Oct 30 01:20:30.737849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:30.737815    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:31.194396 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:31.194347    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:20:31.194613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:31.194406    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 01:20:32.186263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:32.183635    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:20:32.186263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:32.183701    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 01:20:32.396256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:32.395754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:33.189256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:33.188642    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:20:33.189256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:33.188709    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 01:20:34.186266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.179628    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 01:20:34.186266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.179706    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 01:20:34.197881 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.197825    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:34.736426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.736387    4576 generic.go:356] "Generic (PLEG): container finished" podID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerID="84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72" exitCode=0
Oct 30 01:20:34.736676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.736467    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" event={"ID":"dde6c4b4-b01f-4636-8244-26a809e97b77","Type":"ContainerDied","Data":"84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72"}
Oct 30 01:20:34.737054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.737034    4576 scope.go:122] "RemoveContainer" containerID="84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72"
Oct 30 01:20:34.739008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.738982    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:34.739159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.739052    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:34.739159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:34.739086    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:35.238872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.238831    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:20:35.240221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.240194    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:20:35.241350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.241291    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:20:35.745759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.745717    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" event={"ID":"dde6c4b4-b01f-4636-8244-26a809e97b77","Type":"ContainerStarted","Data":"46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508"}
Oct 30 01:20:35.799257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.798525    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"]
Oct 30 01:20:35.802087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.799688    4576 cpu_manager.go:414] "containerMap: removing container" podUID="15aecdd0-cb38-43c1-94f9-dae28674ff13" containerName="revoke-keys"
Oct 30 01:20:35.802087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.799713    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15aecdd0-cb38-43c1-94f9-dae28674ff13" containerName="revoke-keys"
Oct 30 01:20:35.802087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.799898    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="15aecdd0-cb38-43c1-94f9-dae28674ff13" containerName="revoke-keys"
Oct 30 01:20:35.803962 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.803898    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.806222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.804842    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"]
Oct 30 01:20:35.820348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.820323    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:20:35.820757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.820738    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:35.845913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.845881    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.846463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.846427    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwr4s\" (UniqueName: \"kubernetes.io/projected/f8f05c39-782f-4e33-8889-265331db52dd-kube-api-access-qwr4s\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.846668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.846642    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.948119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.948080    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.948367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.948295    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.948448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.948398    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwr4s\" (UniqueName: \"kubernetes.io/projected/f8f05c39-782f-4e33-8889-265331db52dd-kube-api-access-qwr4s\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.948960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.948934    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.951003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.950973    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:35.956738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:35.956713    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:36.032204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:36.032106    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwr4s\" (UniqueName: \"kubernetes.io/projected/f8f05c39-782f-4e33-8889-265331db52dd-kube-api-access-qwr4s\") pod \"maas-api-revoke-sub-73ed64e9-l7lqj\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:36.049482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:36.049447    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:20:36.050997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:36.050963    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 01:20:36.051177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:36.051037    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 01:20:36.121853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:36.121820    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:36.450791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:36.450758    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"]
Oct 30 01:20:36.451346 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:20:36.451322    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f05c39_782f_4e33_8889_265331db52dd.slice/crio-89bf6893e7fe4161a40ec26c31eed0935e727802a996c532c300f0203b37ff43 WatchSource:0}: Error finding container 89bf6893e7fe4161a40ec26c31eed0935e727802a996c532c300f0203b37ff43: Status 404 returned error can't find the container with id 89bf6893e7fe4161a40ec26c31eed0935e727802a996c532c300f0203b37ff43
Oct 30 01:20:36.708980 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:36.708910    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:20:36.809262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:36.805687    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj" event={"ID":"f8f05c39-782f-4e33-8889-265331db52dd","Type":"ContainerStarted","Data":"89bf6893e7fe4161a40ec26c31eed0935e727802a996c532c300f0203b37ff43"}
Oct 30 01:20:37.049450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:37.049417    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 01:20:37.049630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:37.049471    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 01:20:37.491268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:37.489216    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:37.816012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:37.815928    4576 generic.go:356] "Generic (PLEG): container finished" podID="f8f05c39-782f-4e33-8889-265331db52dd" containerID="0c054868c8359f53de5aaa34f844d93f60ad65e12d38066717deed113cd8331d" exitCode=0
Oct 30 01:20:37.816172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:37.816017    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj" event={"ID":"f8f05c39-782f-4e33-8889-265331db52dd","Type":"ContainerDied","Data":"0c054868c8359f53de5aaa34f844d93f60ad65e12d38066717deed113cd8331d"}
Oct 30 01:20:38.049268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:38.049219    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 01:20:38.049457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:38.049285    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 01:20:39.036750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.036721    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:39.049493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.049459    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 01:20:39.049678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.049519    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 01:20:39.091301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.088675    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:20:39.138410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.134128    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-xks-ca\") pod \"f8f05c39-782f-4e33-8889-265331db52dd\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") "
Oct 30 01:20:39.138410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.134202    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-openshift-ca\") pod \"f8f05c39-782f-4e33-8889-265331db52dd\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") "
Oct 30 01:20:39.138410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.135016    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f05c39-782f-4e33-8889-265331db52dd-kube-api-access-qwr4s\" (UniqueName: \"kubernetes.io/projected/f8f05c39-782f-4e33-8889-265331db52dd-kube-api-access-qwr4s\") pod \"f8f05c39-782f-4e33-8889-265331db52dd\" (UID: \"f8f05c39-782f-4e33-8889-265331db52dd\") "
Oct 30 01:20:39.138410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.135375    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-openshift-ca" pod "f8f05c39-782f-4e33-8889-265331db52dd" (UID: "f8f05c39-782f-4e33-8889-265331db52dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:39.138410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.135844    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:39.138410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.137735    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f05c39-782f-4e33-8889-265331db52dd-kube-api-access-qwr4s" pod "f8f05c39-782f-4e33-8889-265331db52dd" (UID: "f8f05c39-782f-4e33-8889-265331db52dd"). InnerVolumeSpecName "kube-api-access-qwr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:39.138410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.137813    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-xks-ca" pod "f8f05c39-782f-4e33-8889-265331db52dd" (UID: "f8f05c39-782f-4e33-8889-265331db52dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:39.238345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.238307    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f05c39-782f-4e33-8889-265331db52dd-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:39.238345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.238332    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwr4s\" (UniqueName: \"kubernetes.io/projected/f8f05c39-782f-4e33-8889-265331db52dd-kube-api-access-qwr4s\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:20:39.456112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.456079    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:39.834565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.834489    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"
Oct 30 01:20:39.834565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.834514    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj" event={"ID":"f8f05c39-782f-4e33-8889-265331db52dd","Type":"ContainerDied","Data":"89bf6893e7fe4161a40ec26c31eed0935e727802a996c532c300f0203b37ff43"}
Oct 30 01:20:39.834756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:39.834566    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89bf6893e7fe4161a40ec26c31eed0935e727802a996c532c300f0203b37ff43"
Oct 30 01:20:40.039676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:40.039638    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:40.048928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:40.048900    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 01:20:40.049093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:40.048944    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 01:20:41.049555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:41.049518    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 01:20:41.050057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:41.049587    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 01:20:41.080268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:41.080214    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:41.666106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:41.666075    4576 scope.go:122] "RemoveContainer" containerID="9a553e082107de2688e537bf04b862177cb0346e953d45b24c4c0b4899f3b8f2"
Oct 30 01:20:41.676739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:41.676718    4576 scope.go:122] "RemoveContainer" containerID="fabb659117c8c6e9ec2af4cca771906a7050684b51ae10ade5d4c71d3308019f"
Oct 30 01:20:42.053701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:42.053672    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:20:42.064011 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:42.063980    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:20:42.771205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:42.771165    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:43.605747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:43.605705    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:44.394075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:44.394037    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:44.504467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:44.504425    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:45.051464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.051426    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"]
Oct 30 01:20:45.052269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.052248    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f8f05c39-782f-4e33-8889-265331db52dd" containerName="revoke-keys"
Oct 30 01:20:45.052424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.052271    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f05c39-782f-4e33-8889-265331db52dd" containerName="revoke-keys"
Oct 30 01:20:45.052502 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.052489    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f05c39-782f-4e33-8889-265331db52dd" containerName="revoke-keys"
Oct 30 01:20:45.057625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.057607    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.088167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.088137    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-24d5ce-openshift-default-dockercfg-bwdz5\""
Oct 30 01:20:45.092865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.092838    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"]
Oct 30 01:20:45.229349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.229315    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b3754-1cef-4071-b077-2663fcd55336-istiod-ca-cert\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.229577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.229523    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-credential-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.229683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.229661    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8df7\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-kube-api-access-c8df7\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.229744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.229709    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-istio-token\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.229826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.229813    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.229869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.229847    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-certs\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.229930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.229880    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-envoy\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.229986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.229949    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-data\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.230095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.230068    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b3754-1cef-4071-b077-2663fcd55336-istio-podinfo\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.331897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.331810    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b3754-1cef-4071-b077-2663fcd55336-istiod-ca-cert\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.332250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332209    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-credential-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.332380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332320    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8df7\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-kube-api-access-c8df7\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.332380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332351    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-istio-token\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.332510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332420    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.332510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332464    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-certs\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.332623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332514    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-envoy\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.332623 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332540    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-data\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.333051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332624    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b3754-1cef-4071-b077-2663fcd55336-istio-podinfo\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.333051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332630    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-credential-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.333051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332669    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b3754-1cef-4071-b077-2663fcd55336-istiod-ca-cert\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.333051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.332852    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-certs\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.333329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.333061    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-socket\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.333329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.333104    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-data\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.335385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.335344    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-envoy\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.335483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.335413    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b3754-1cef-4071-b077-2663fcd55336-istio-podinfo\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.353335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.353308    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-istio-token\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.353450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.353439    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8df7\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-kube-api-access-c8df7\") pod \"e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") " pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.371795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.371776    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:45.690861 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:20:45.690828    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900b3754_1cef_4071_b077_2663fcd55336.slice/crio-d53df7cc0b372fd15c77f132ab0705f2a8e66211093c05b21b11aa2d3ef8c093 WatchSource:0}: Error finding container d53df7cc0b372fd15c77f132ab0705f2a8e66211093c05b21b11aa2d3ef8c093: Status 404 returned error can't find the container with id d53df7cc0b372fd15c77f132ab0705f2a8e66211093c05b21b11aa2d3ef8c093
Oct 30 01:20:45.691511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.691486    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"]
Oct 30 01:20:45.693808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.693779    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:45.693874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.693842    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:45.693956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.693871    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:20:45.882686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.882648    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" event={"ID":"900b3754-1cef-4071-b077-2663fcd55336","Type":"ContainerStarted","Data":"3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40"}
Oct 30 01:20:45.882896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.882713    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" event={"ID":"900b3754-1cef-4071-b077-2663fcd55336","Type":"ContainerStarted","Data":"d53df7cc0b372fd15c77f132ab0705f2a8e66211093c05b21b11aa2d3ef8c093"}
Oct 30 01:20:45.929569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.929502    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podStartSLOduration=0.929487808 podStartE2EDuration="929.487808ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:45.927690429 +0000 UTC m=+2525.999014276" watchObservedRunningTime="2026-10-30 01:20:45.929487808 +0000 UTC m=+2526.000811743"
Oct 30 01:20:45.943887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:45.943793    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:46.372569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:46.372487    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:46.374368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:46.374341    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 01:20:46.374504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:46.374395    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 01:20:47.383259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:47.380584    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 01:20:47.383259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:47.380661    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 01:20:47.624152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:47.624116    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:47.817598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:47.817563    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:48.011380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:48.011350    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:48.399257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:48.399092    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 01:20:48.399257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:48.399155    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 01:20:48.454044 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:20:48.454007    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:20:49.377605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:49.377571    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 01:20:49.377905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:49.377869    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 01:20:49.933173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:49.933129    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:50.387684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:50.387645    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 01:20:50.387998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:50.387968    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 01:20:51.392106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:51.391632    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 01:20:51.392106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:51.391700    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 01:20:51.844519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:51.844482    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:52.372419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:52.372385    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 01:20:52.372717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:52.372692    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 01:20:53.363472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:53.363437    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:53.378775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:53.378747    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:53.379284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:53.379265    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:53.380137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:53.380117    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:20:54.803613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:54.803570    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:55.029409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:55.029374    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"]
Oct 30 01:20:55.043058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:55.043024    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5602db-bjs42"]
Oct 30 01:20:56.111348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:56.111316    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d6205c-1d2f-4325-a056-66442ca51f58" path="/var/lib/kubelet/pods/96d6205c-1d2f-4325-a056-66442ca51f58/volumes"
Oct 30 01:20:56.452851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:56.452819    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:57.902080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:57.902041    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:59.355453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:59.355414    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:20:59.424059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:59.424024    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"]
Oct 30 01:20:59.490143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:20:59.490108    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:00.091249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:00.091180    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:00.538995 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:00.538958    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:00.702840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:00.702806    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:01.805093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:01.805055    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:03.002626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:03.002586    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:03.029831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:03.029797    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"]
Oct 30 01:21:03.043956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:03.043924    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863b7fe1-g7k6j"]
Oct 30 01:21:03.224164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:03.224133    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:04.112784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:04.112755    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02389651-5192-43ce-b05b-43f5a6a484ca" path="/var/lib/kubelet/pods/02389651-5192-43ce-b05b-43f5a6a484ca/volumes"
Oct 30 01:21:04.692926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:04.692780    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:04.830754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:04.830727    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:04.948191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:04.948078    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:06.501499 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.501460    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:06.620503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.620449    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:06.702455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.702416    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"]
Oct 30 01:21:06.711481 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.711448    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"]
Oct 30 01:21:06.711814 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.711792    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:06.729182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.729132    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-11fbe8\""
Oct 30 01:21:06.822926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.818761    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-11fbe8-dockercfg-s8845\""
Oct 30 01:21:06.848326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.839422    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ksf\" (UniqueName: \"kubernetes.io/projected/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-kube-api-access-w5ksf\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8\" (UID: \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:06.947312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.848810    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8\" (UID: \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:06.947983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.947315    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"]
Oct 30 01:21:06.966524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.966488    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"]
Oct 30 01:21:06.966721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:06.966553    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:07.054367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.052357    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ksf\" (UniqueName: \"kubernetes.io/projected/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-kube-api-access-w5ksf\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8\" (UID: \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:07.054367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.052482    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8\" (UID: \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:07.054367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.053347    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8\" (UID: \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:07.073712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.073633    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ksf\" (UniqueName: \"kubernetes.io/projected/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-kube-api-access-w5ksf\") pod \"payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8\" (UID: \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:07.156524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.156483    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:07.156750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.156732    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2570a7e8-ff3b-4b05-a333-08cec97a0363-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6\" (UID: \"2570a7e8-ff3b-4b05-a333-08cec97a0363\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:07.156867 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.156792    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ctt\" (UniqueName: \"kubernetes.io/projected/2570a7e8-ff3b-4b05-a333-08cec97a0363-kube-api-access-42ctt\") pod \"payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6\" (UID: \"2570a7e8-ff3b-4b05-a333-08cec97a0363\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:07.201666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.201629    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"]
Oct 30 01:21:07.234877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.234846    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"]
Oct 30 01:21:07.258208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.258172    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ctt\" (UniqueName: \"kubernetes.io/projected/2570a7e8-ff3b-4b05-a333-08cec97a0363-kube-api-access-42ctt\") pod \"payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6\" (UID: \"2570a7e8-ff3b-4b05-a333-08cec97a0363\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:07.258536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.258484    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2570a7e8-ff3b-4b05-a333-08cec97a0363-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6\" (UID: \"2570a7e8-ff3b-4b05-a333-08cec97a0363\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:07.259279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.259217    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2570a7e8-ff3b-4b05-a333-08cec97a0363-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6\" (UID: \"2570a7e8-ff3b-4b05-a333-08cec97a0363\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:07.279408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.279380    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ctt\" (UniqueName: \"kubernetes.io/projected/2570a7e8-ff3b-4b05-a333-08cec97a0363-kube-api-access-42ctt\") pod \"payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6\" (UID: \"2570a7e8-ff3b-4b05-a333-08cec97a0363\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:07.336840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.336753    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:07.365808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.364981    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:07.375327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.366161    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:07.482038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.478681    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:07.674883 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:07.674846    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f77b7f_4d8b_47ea_bc75_5b3d70a63b05.slice/crio-22b6b3c2be6910c420fb297bf5dadb77c69749b7a4816b7a3494ec9b9f01f3fd WatchSource:0}: Error finding container 22b6b3c2be6910c420fb297bf5dadb77c69749b7a4816b7a3494ec9b9f01f3fd: Status 404 returned error can't find the container with id 22b6b3c2be6910c420fb297bf5dadb77c69749b7a4816b7a3494ec9b9f01f3fd
Oct 30 01:21:07.675448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.675424    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"]
Oct 30 01:21:07.702305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:07.702276    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"]
Oct 30 01:21:07.702499 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:07.702390    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2570a7e8_ff3b_4b05_a333_08cec97a0363.slice/crio-f3d27a6eaf9564c8198df2db6006d1d708068553b366c2e3838d841c013727a0 WatchSource:0}: Error finding container f3d27a6eaf9564c8198df2db6006d1d708068553b366c2e3838d841c013727a0: Status 404 returned error can't find the container with id f3d27a6eaf9564c8198df2db6006d1d708068553b366c2e3838d841c013727a0
Oct 30 01:21:08.142601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.142566    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6" event={"ID":"2570a7e8-ff3b-4b05-a333-08cec97a0363","Type":"ContainerStarted","Data":"6854cbfd2573a62c29784ab219dbcd28bbb9fb021cbbc358c560bee3b1cf03c8"}
Oct 30 01:21:08.142807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.142618    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6" event={"ID":"2570a7e8-ff3b-4b05-a333-08cec97a0363","Type":"ContainerStarted","Data":"f3d27a6eaf9564c8198df2db6006d1d708068553b366c2e3838d841c013727a0"}
Oct 30 01:21:08.142807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.142670    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:08.145359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.144725    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8" event={"ID":"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05","Type":"ContainerStarted","Data":"e54e05199ecab15f1bba9048fcbf822d484da479b3a492bb0d254aab0e2fdc72"}
Oct 30 01:21:08.145359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.144774    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8" event={"ID":"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05","Type":"ContainerStarted","Data":"22b6b3c2be6910c420fb297bf5dadb77c69749b7a4816b7a3494ec9b9f01f3fd"}
Oct 30 01:21:08.145359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.144900    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:08.194674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.194614    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6" podStartSLOduration=2.194594957 podStartE2EDuration="2.194594957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:08.19270208 +0000 UTC m=+2548.264025939" watchObservedRunningTime="2026-10-30 01:21:08.194594957 +0000 UTC m=+2548.265918793"
Oct 30 01:21:08.239582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.239527    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8" podStartSLOduration=2.239512166 podStartE2EDuration="2.239512166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:08.234142161 +0000 UTC m=+2548.305465996" watchObservedRunningTime="2026-10-30 01:21:08.239512166 +0000 UTC m=+2548.310836015"
Oct 30 01:21:08.502695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:08.502657    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:09.565854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:09.565813    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:09.682627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:09.682575    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:09.709971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:09.709942    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"]
Oct 30 01:21:09.723277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:09.723183    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-283eb6-pbcfj"]
Oct 30 01:21:09.803552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:09.803518    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:10.112761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:10.112727    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7" path="/var/lib/kubelet/pods/ac26ca1c-3ab2-49ec-a5ec-8b3f03cefdb7/volumes"
Oct 30 01:21:10.159533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:10.159500    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:10.378301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:10.378209    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:10.485375 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:10.485331    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:11.140463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.140425    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"]
Oct 30 01:21:11.146951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.146925    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.169500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.169470    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"]
Oct 30 01:21:11.174432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.174405    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-60f38f\"/\"kube-root-ca.crt\""
Oct 30 01:21:11.175129 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.175101    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-60f38f\"/\"openshift-service-ca.crt\""
Oct 30 01:21:11.178723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.178693    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-60f38f\"/\"default-dockercfg-fc5sd\""
Oct 30 01:21:11.178851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.178766    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-60f38f\"/\"auth-test-model-60f38f-kserve-self-signed-certs\""
Oct 30 01:21:11.225692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.225658    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:11.271346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.271312    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2vw\" (UniqueName: \"kubernetes.io/projected/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-kube-api-access-nb2vw\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.271548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.271365    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-model-cache\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.271548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.271448    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-home\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.271638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.271568    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tmp-dir\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.271844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.271823    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-dshm\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.271889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.271872    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tls-certs\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.386474    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"]
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.390382    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-model-cache\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.390434    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-home\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.390501    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tmp-dir\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.390687    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-dshm\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.390725    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tls-certs\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.390889    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2vw\" (UniqueName: \"kubernetes.io/projected/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-kube-api-access-nb2vw\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.391672    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-model-cache\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.391935    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-home\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.395302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.392167    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tmp-dir\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.420264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.410066    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.430681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.422562    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"]
Oct 30 01:21:11.499552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.496366    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.499552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.496472    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.499552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.496521    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmv2v\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-kube-api-access-wmv2v\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.499552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.496565    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.499552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.496635    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.508271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.502005    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.508271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.502131    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.508271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.502267    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.508271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.502343    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.508271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.505666    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-dshm\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.533937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.524244    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tls-certs\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.577331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.576674    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2vw\" (UniqueName: \"kubernetes.io/projected/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-kube-api-access-nb2vw\") pod \"auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") " pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.615500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.614998    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.615500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.615090    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.618612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.616126    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.618612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.617358    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.618612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.617441    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.618612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.617469    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmv2v\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-kube-api-access-wmv2v\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.618612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.617499    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.618612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.617545    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.618612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.617575    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.618612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.617636    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.619202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.618980    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.619202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.619161    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.646385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.620706    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.646385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.620966    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.646385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.622151    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.646385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.623514    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.646385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.624211    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:11.658277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.658220    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.670565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.670525    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmv2v\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-kube-api-access-wmv2v\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:11.771358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.771323    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:11.808065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:11.808035    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:12.082251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.082209    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"]
Oct 30 01:21:12.082646 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:12.082621    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a8c0f87_19cd_4bf9_8e32_cc907bfb118b.slice/crio-0fd350a2b496ed7c00e7153707afe19c58f543e0e5314dbe328e3d01578c80a9 WatchSource:0}: Error finding container 0fd350a2b496ed7c00e7153707afe19c58f543e0e5314dbe328e3d01578c80a9: Status 404 returned error can't find the container with id 0fd350a2b496ed7c00e7153707afe19c58f543e0e5314dbe328e3d01578c80a9
Oct 30 01:21:12.151487 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:12.151453    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67ef6a4_517e_4b89_97f3_e8800f0cbefc.slice/crio-9ee076467e159418b06e18696adec4824875fd086c4a0d4ffcb112099bf93b68 WatchSource:0}: Error finding container 9ee076467e159418b06e18696adec4824875fd086c4a0d4ffcb112099bf93b68: Status 404 returned error can't find the container with id 9ee076467e159418b06e18696adec4824875fd086c4a0d4ffcb112099bf93b68
Oct 30 01:21:12.152601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.152573    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"]
Oct 30 01:21:12.154807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.154774    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:12.154893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.154846    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:12.154893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.154875    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:12.202123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.202088    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb" podUID="07a9b46f-d980-4465-9dcf-158a3b2d9de0" containerName="payload-processing" containerID="cri-o://6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34" gracePeriod=25
Oct 30 01:21:12.204356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.204331    4576 generic.go:356] "Generic (PLEG): container finished" podID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerID="c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f" exitCode=0
Oct 30 01:21:12.204492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.204414    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" event={"ID":"289432fe-271c-4f17-8ae3-edc5b8794d46","Type":"ContainerDied","Data":"c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f"}
Oct 30 01:21:12.205196 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.205174    4576 scope.go:122] "RemoveContainer" containerID="c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f"
Oct 30 01:21:12.206719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.206685    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" event={"ID":"f67ef6a4-517e-4b89-97f3-e8800f0cbefc","Type":"ContainerStarted","Data":"9ee076467e159418b06e18696adec4824875fd086c4a0d4ffcb112099bf93b68"}
Oct 30 01:21:12.207733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.207706    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:12.207868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.207796    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:12.207868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.207847    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:12.208890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.208861    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm" event={"ID":"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b","Type":"ContainerStarted","Data":"299c4209eaebd6113a7d20be097c40ab900d789889da0b7bcc2ef309e7952d37"}
Oct 30 01:21:12.208983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.208905    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm" event={"ID":"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b","Type":"ContainerStarted","Data":"0fd350a2b496ed7c00e7153707afe19c58f543e0e5314dbe328e3d01578c80a9"}
Oct 30 01:21:12.236903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.235492    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p" podUID="93f06dd1-fc1b-498a-8658-5639c6bd0951" containerName="payload-pre-processing" containerID="cri-o://3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4" gracePeriod=25
Oct 30 01:21:12.242309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.242279    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"]
Oct 30 01:21:12.246895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.246872    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.272122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.272091    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"]
Oct 30 01:21:12.278183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.278149    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"]
Oct 30 01:21:12.283076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.283052    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.303455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.299757    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"]
Oct 30 01:21:12.334955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.334918    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.334967    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.335097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335086    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.335279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335178    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.335279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335209    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkv65\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-kube-api-access-wkv65\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.335413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335277    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335308    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335380    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335408    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335435    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335493    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335535    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.335809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335642    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.335809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335705    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.335809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335760    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9026ba9-d689-41e5-b5df-f0371f725e9d-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.335984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335823    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b572d61-32a8-4ca9-aa31-0640535d2897-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335847    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62cv\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-kube-api-access-s62cv\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.335984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.335895    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.364063 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.363983    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm" podStartSLOduration=1.363961766 podStartE2EDuration="1.363961766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:12.309069927 +0000 UTC m=+2552.380393763" watchObservedRunningTime="2026-10-30 01:21:12.363961766 +0000 UTC m=+2552.435285605"
Oct 30 01:21:12.445984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.445841    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9026ba9-d689-41e5-b5df-f0371f725e9d-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.452771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.452723    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9026ba9-d689-41e5-b5df-f0371f725e9d-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.453065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.453027    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b572d61-32a8-4ca9-aa31-0640535d2897-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.454631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.454610    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62cv\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-kube-api-access-s62cv\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.455301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.455269    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.455906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.455874    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.456802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.456755    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.457165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.457141    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.458101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.458079    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.458319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.458299    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkv65\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-kube-api-access-wkv65\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.459099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.457946    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.459099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.454447    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b572d61-32a8-4ca9-aa31-0640535d2897-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.459099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.455746    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.459755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.459691    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.459884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.458914    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.460031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.460015    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.460618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.460599    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.462991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.460923    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.463174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.463153    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.463757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.463726    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.464280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.464262    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.464872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.464841    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.465513 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.465495    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.466345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.464682    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.466515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.461305    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.466641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.464152    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.466798 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.466112    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.466917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.463626    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.478106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.478026    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.479379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.479323    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.479993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.479949    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.480359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.480319    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.491730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.491658    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.512788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.506731    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62cv\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-kube-api-access-s62cv\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.512788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.510658    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.512788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.511746    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkv65\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-kube-api-access-wkv65\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.630583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.630247    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:21:12.638649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.638430    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:12.656716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.656492    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:12.717354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.716882    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:21:12.772380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.772317    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:12.793871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.793824    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a9b46f-d980-4465-9dcf-158a3b2d9de0-kube-api-access-xbzjw\" (UniqueName: \"kubernetes.io/projected/07a9b46f-d980-4465-9dcf-158a3b2d9de0-kube-api-access-xbzjw\") pod \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\" (UID: \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\") "
Oct 30 01:21:12.794022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.793970    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f06dd1-fc1b-498a-8658-5639c6bd0951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f06dd1-fc1b-498a-8658-5639c6bd0951-plugins-config-volume\") pod \"93f06dd1-fc1b-498a-8658-5639c6bd0951\" (UID: \"93f06dd1-fc1b-498a-8658-5639c6bd0951\") "
Oct 30 01:21:12.794096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.794017    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f06dd1-fc1b-498a-8658-5639c6bd0951-kube-api-access-cc7xx\" (UniqueName: \"kubernetes.io/projected/93f06dd1-fc1b-498a-8658-5639c6bd0951-kube-api-access-cc7xx\") pod \"93f06dd1-fc1b-498a-8658-5639c6bd0951\" (UID: \"93f06dd1-fc1b-498a-8658-5639c6bd0951\") "
Oct 30 01:21:12.794369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.794345    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a9b46f-d980-4465-9dcf-158a3b2d9de0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a9b46f-d980-4465-9dcf-158a3b2d9de0-plugins-config-volume\") pod \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\" (UID: \"07a9b46f-d980-4465-9dcf-158a3b2d9de0\") "
Oct 30 01:21:12.802127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.802091    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f06dd1-fc1b-498a-8658-5639c6bd0951-plugins-config-volume" pod "93f06dd1-fc1b-498a-8658-5639c6bd0951" (UID: "93f06dd1-fc1b-498a-8658-5639c6bd0951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:12.802550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.802502    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a9b46f-d980-4465-9dcf-158a3b2d9de0-plugins-config-volume" pod "07a9b46f-d980-4465-9dcf-158a3b2d9de0" (UID: "07a9b46f-d980-4465-9dcf-158a3b2d9de0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:12.804142 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.804085    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f06dd1-fc1b-498a-8658-5639c6bd0951-kube-api-access-cc7xx" pod "93f06dd1-fc1b-498a-8658-5639c6bd0951" (UID: "93f06dd1-fc1b-498a-8658-5639c6bd0951"). InnerVolumeSpecName "kube-api-access-cc7xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:12.805568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.805542    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a9b46f-d980-4465-9dcf-158a3b2d9de0-kube-api-access-xbzjw" pod "07a9b46f-d980-4465-9dcf-158a3b2d9de0" (UID: "07a9b46f-d980-4465-9dcf-158a3b2d9de0"). InnerVolumeSpecName "kube-api-access-xbzjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:12.897540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.897428    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a9b46f-d980-4465-9dcf-158a3b2d9de0-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:12.897540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.897467    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbzjw\" (UniqueName: \"kubernetes.io/projected/07a9b46f-d980-4465-9dcf-158a3b2d9de0-kube-api-access-xbzjw\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:12.897540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.897484    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f06dd1-fc1b-498a-8658-5639c6bd0951-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:12.897540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:12.897499    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc7xx\" (UniqueName: \"kubernetes.io/projected/93f06dd1-fc1b-498a-8658-5639c6bd0951-kube-api-access-cc7xx\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:13.012472 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:13.012433    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9026ba9_d689_41e5_b5df_f0371f725e9d.slice/crio-82cfeaf1db8c90b63576929ff0a60c40957ca4d058736776b7c20dbb0189d89d WatchSource:0}: Error finding container 82cfeaf1db8c90b63576929ff0a60c40957ca4d058736776b7c20dbb0189d89d: Status 404 returned error can't find the container with id 82cfeaf1db8c90b63576929ff0a60c40957ca4d058736776b7c20dbb0189d89d
Oct 30 01:21:13.016089 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.016042    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"]
Oct 30 01:21:13.017651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.017609    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:13.018487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.018453    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:13.018691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.018660    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:13.030031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.027880    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"]
Oct 30 01:21:13.030993 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:13.030477    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b572d61_32a8_4ca9_aa31_0640535d2897.slice/crio-532b74196aa3a5eea6ff6d4f6658f60e8a67f284899b064c53eea4dc7b1f5cae WatchSource:0}: Error finding container 532b74196aa3a5eea6ff6d4f6658f60e8a67f284899b064c53eea4dc7b1f5cae: Status 404 returned error can't find the container with id 532b74196aa3a5eea6ff6d4f6658f60e8a67f284899b064c53eea4dc7b1f5cae
Oct 30 01:21:13.035089 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.035049    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:13.035211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.035136    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:13.035211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.035186    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:13.255580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.254747    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" event={"ID":"289432fe-271c-4f17-8ae3-edc5b8794d46","Type":"ContainerStarted","Data":"cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585"}
Oct 30 01:21:13.260534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.260406    4576 generic.go:356] "Generic (PLEG): container finished" podID="93f06dd1-fc1b-498a-8658-5639c6bd0951" containerID="3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4" exitCode=0
Oct 30 01:21:13.260846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.260813    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p" event={"ID":"93f06dd1-fc1b-498a-8658-5639c6bd0951","Type":"ContainerDied","Data":"3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4"}
Oct 30 01:21:13.261096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.261078    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p" event={"ID":"93f06dd1-fc1b-498a-8658-5639c6bd0951","Type":"ContainerDied","Data":"1d6b45dcf5ffa67499e2eef4ee58e8044d50852ce89a3753760cdc1bfb8b9fa1"}
Oct 30 01:21:13.261277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.261262    4576 scope.go:122] "RemoveContainer" containerID="3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4"
Oct 30 01:21:13.262323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.262301    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"
Oct 30 01:21:13.309648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.309600    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" event={"ID":"4b572d61-32a8-4ca9-aa31-0640535d2897","Type":"ContainerStarted","Data":"f61e8926577dd07159887fe96849405224c5cd143bd38fd081aa8bd7802859be"}
Oct 30 01:21:13.309964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.309926    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" event={"ID":"4b572d61-32a8-4ca9-aa31-0640535d2897","Type":"ContainerStarted","Data":"532b74196aa3a5eea6ff6d4f6658f60e8a67f284899b064c53eea4dc7b1f5cae"}
Oct 30 01:21:13.328058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.328004    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" event={"ID":"a9026ba9-d689-41e5-b5df-f0371f725e9d","Type":"ContainerStarted","Data":"82cfeaf1db8c90b63576929ff0a60c40957ca4d058736776b7c20dbb0189d89d"}
Oct 30 01:21:13.345328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.345309    4576 scope.go:122] "RemoveContainer" containerID="3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4"
Oct 30 01:21:13.345822 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:21:13.345802    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4\": container with ID starting with 3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4 not found: ID does not exist" containerID="3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4"
Oct 30 01:21:13.345893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.345832    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4"} err="failed to get container status \"3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4\": rpc error: code = NotFound desc = could not find container \"3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4\": container with ID starting with 3d3b4524d49e3990b2b7fca5aa89242b0b66993d50d62b95ecf48e8c062f93b4 not found: ID does not exist"
Oct 30 01:21:13.346039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.346022    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" event={"ID":"f67ef6a4-517e-4b89-97f3-e8800f0cbefc","Type":"ContainerStarted","Data":"56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac"}
Oct 30 01:21:13.355283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.349584    4576 generic.go:356] "Generic (PLEG): container finished" podID="07a9b46f-d980-4465-9dcf-158a3b2d9de0" containerID="6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34" exitCode=0
Oct 30 01:21:13.355283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.350305    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"
Oct 30 01:21:13.355283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.354831    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb" event={"ID":"07a9b46f-d980-4465-9dcf-158a3b2d9de0","Type":"ContainerDied","Data":"6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34"}
Oct 30 01:21:13.355283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.354988    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb" event={"ID":"07a9b46f-d980-4465-9dcf-158a3b2d9de0","Type":"ContainerDied","Data":"3642d95933d92c90b3bd7c1d27b7f0f1d7393fa9a33561687d45711591b30a03"}
Oct 30 01:21:13.355283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.355021    4576 scope.go:122] "RemoveContainer" containerID="6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34"
Oct 30 01:21:13.400326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.400261    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podStartSLOduration=1.400207417 podStartE2EDuration="1.400207417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:13.399605816 +0000 UTC m=+2553.470929652" watchObservedRunningTime="2026-10-30 01:21:13.400207417 +0000 UTC m=+2553.471531253"
Oct 30 01:21:13.433930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.433899    4576 scope.go:122] "RemoveContainer" containerID="6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34"
Oct 30 01:21:13.451137 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:21:13.451100    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34\": container with ID starting with 6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34 not found: ID does not exist" containerID="6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34"
Oct 30 01:21:13.452529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.452472    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34"} err="failed to get container status \"6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34\": rpc error: code = NotFound desc = could not find container \"6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34\": container with ID starting with 6ec30b6c72097b0f68ff42ec67d0763214ede03b18354ae40f56e884d9116b34 not found: ID does not exist"
Oct 30 01:21:13.455275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.454362    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"]
Oct 30 01:21:13.459152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.459128    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-07a3a8-6c57cb9bc8-wnxxb"]
Oct 30 01:21:13.538832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.538700    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podStartSLOduration=2.538680574 podStartE2EDuration="2.538680574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:13.516392686 +0000 UTC m=+2553.587716521" watchObservedRunningTime="2026-10-30 01:21:13.538680574 +0000 UTC m=+2553.610004410"
Oct 30 01:21:13.564253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.564205    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"]
Oct 30 01:21:13.588968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.588926    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-07a3a8-5954bb7b87-s5k4p"]
Oct 30 01:21:13.651161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.651115    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:13.657454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.657424    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:13.703260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.702968    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:21:13.721262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.718531    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:21:13.721262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.718593    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 01:21:13.721262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.720510    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:13.810316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.808602    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:13.821769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.820425    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:21:13.821769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:13.820505    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 01:21:14.038525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.038399    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podStartSLOduration=2.038381985 podStartE2EDuration="2.038381985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:13.631591386 +0000 UTC m=+2553.702915222" watchObservedRunningTime="2026-10-30 01:21:14.038381985 +0000 UTC m=+2554.109705817"
Oct 30 01:21:14.042605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.042567    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"]
Oct 30 01:21:14.046207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.046176    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd6ac37b-bs5vk"]
Oct 30 01:21:14.157578 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.157490    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a9b46f-d980-4465-9dcf-158a3b2d9de0" path="/var/lib/kubelet/pods/07a9b46f-d980-4465-9dcf-158a3b2d9de0/volumes"
Oct 30 01:21:14.161008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.160961    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eecacef-9738-4ee8-b88a-55d582baf4a0" path="/var/lib/kubelet/pods/5eecacef-9738-4ee8-b88a-55d582baf4a0/volumes"
Oct 30 01:21:14.161971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.161930    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f06dd1-fc1b-498a-8658-5639c6bd0951" path="/var/lib/kubelet/pods/93f06dd1-fc1b-498a-8658-5639c6bd0951/volumes"
Oct 30 01:21:14.330388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.330348    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:21:14.444320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.430468    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" event={"ID":"a9026ba9-d689-41e5-b5df-f0371f725e9d","Type":"ContainerStarted","Data":"a9764b3dc22e97996afce1237aef149798e7f66ca8d2153bc2306a2d3ca915f7"}
Oct 30 01:21:14.594258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.593301    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main" containerID="cri-o://696ccb1d5cd705629e6473626eabcf6d7f28886f84c579240349e10a06d362b4" gracePeriod=45
Oct 30 01:21:14.656870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.656836    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:14.657521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.657158    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:14.669462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.669422    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:21:14.669773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.669727    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 01:21:14.712510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.712410    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.181:8000/ready\": read tcp 10.133.0.2:37512->10.133.0.181:8000: read: connection reset by peer"
Oct 30 01:21:14.719476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.719439    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:21:14.719730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.719707    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 01:21:14.809069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.809022    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:21:14.809426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:14.809380    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 01:21:15.365825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.365781    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:15.398119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.398073    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:15.547272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.546713    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.181:8000/ready\": dial tcp 10.133.0.181:8000: connect: connection refused"
Oct 30 01:21:15.547272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.546841    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:21:15.599737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.599332    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:15.652456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.652366    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:15.652718 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.652686    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:15.655659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.655314    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:15.662268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.660958    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:21:15.662268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.661005    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 01:21:15.679903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.679865    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:15.705197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.705154    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:21:15.705534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.705490    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 01:21:15.764272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.762454    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:15.839257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.839070    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:21:15.839257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.839146    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 01:21:15.933480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:15.933362    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:16.037617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.037571    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:16.065801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.037900    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:16.092870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.092826    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:16.117376 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.117323    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": context deadline exceeded"
Oct 30 01:21:16.118651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.117679    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:16.131400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.131338    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:16.298723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.298684    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:16.390261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.373408    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:16.390261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.374158    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 01:21:16.479426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.478640    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:16.604318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.583826    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.181:8000/ready\": dial tcp 10.133.0.181:8000: connect: connection refused"
Oct 30 01:21:16.605109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.605070    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 01:21:16.614568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.611704    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 01:21:16.614568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.605347    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 01:21:16.700667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.641677    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:16.700667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.641735    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:16.700667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.676710    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:16.700667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.677119    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:21:16.713004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.701158    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:21:16.713004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.702847    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:21:16.713004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.702891    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 01:21:16.713663 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.701216    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 01:21:16.812257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.812202    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:21:16.812547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.812516    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 01:21:16.886819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:16.886579    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:17.022954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.022914    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.023515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.023491    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 01:21:17.079270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.067839    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.079270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.068206    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 01:21:17.079270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.067376    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.119261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.094353    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.119261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.100129    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 01:21:17.119261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.100207    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 01:21:17.122441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.122398    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.268062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.151729    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.269526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.194019    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"]
Oct 30 01:21:17.269794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.269777    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:21:17.269968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.269955    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 01:21:17.271296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.271270    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy" containerID="cri-o://ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd" gracePeriod=30
Oct 30 01:21:17.290198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.290101    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:21:17.330511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.329082    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.365389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.365347    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.367016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.366990    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 01:21:17.408283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.392787    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 01:21:17.551399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.551312    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.181:8000/ready\": dial tcp 10.133.0.181:8000: connect: connection refused"
Oct 30 01:21:17.581993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.579816    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 01:21:17.662276 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:21:17.648083    4576 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddbd16dd_6a19_42e2_9406_fc4d963a53c2.slice/crio-conmon-696ccb1d5cd705629e6473626eabcf6d7f28886f84c579240349e10a06d362b4.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:21:17.662276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.652315    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 01:21:17.693410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.663022    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.693410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.668454    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 01:21:17.693410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.669334    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:21:17.693410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.669382    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:21:17.693410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.676993    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:17.724743 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.724705    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:21:17.724993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.724968    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 01:21:17.740255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.725141    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:17.740584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.740548    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:17.751261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.747967    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:17.777391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.768717    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:21:17.777703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.777672    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 01:21:17.777855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.769700    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 01:21:17.880368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.880285    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:21:17.880624 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:17.880591    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 01:21:18.086689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.071013    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:18.101653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.083275    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:18.139783 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.096713    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:18.180497 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.113569    4576 generic.go:356] "Generic (PLEG): container finished" podID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerID="696ccb1d5cd705629e6473626eabcf6d7f28886f84c579240349e10a06d362b4" exitCode=0
Oct 30 01:21:18.342503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.341897    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:18.424372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.369170    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:18.426485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.426410    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" event={"ID":"ddbd16dd-6a19-42e2-9406-fc4d963a53c2","Type":"ContainerDied","Data":"696ccb1d5cd705629e6473626eabcf6d7f28886f84c579240349e10a06d362b4"}
Oct 30 01:21:18.458215 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.451282    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:18.566723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.560413    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:18.847494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:18.847449    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.203831    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.205088    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.206280    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": context deadline exceeded"
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.228961    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.229056    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.229194    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.229254    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.229347    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.229379    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.229470    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.235402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.229522    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 01:21:19.257474 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:21:19.250015    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:21:19.328260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.319207    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 01:21:19.328260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.319345    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:21:19.348773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.345902    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:19.431345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.381470    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:19.446264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.433304    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 01:21:19.446264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.439351    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 01:21:19.541388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.518364    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:19.574656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.570214    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:19.603256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.580888    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:19.631318 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.601459    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:19.662603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.603747    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.181:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:19.663108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.663078    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:21:19.843506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.843390    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.843821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.843776    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:19.884491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.884454    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.884762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.884735    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 01:21:19.884983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.884965    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.885104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.885087    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 01:21:19.962875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.962821    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 01:21:19.963185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:19.963157    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 01:21:20.105310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.102744    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn" event={"ID":"ddbd16dd-6a19-42e2-9406-fc4d963a53c2","Type":"ContainerDied","Data":"4240cfbe0a86d3747e97cff679fe5fb67523ce8954f4fbf0f5b72caffc8c272d"}
Oct 30 01:21:20.105310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.102795    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4240cfbe0a86d3747e97cff679fe5fb67523ce8954f4fbf0f5b72caffc8c272d"
Oct 30 01:21:20.130511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.130437    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:21:20.150392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.150358    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:20.170801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.156784    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"]
Oct 30 01:21:20.217821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.217430    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.225258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.217547    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.239321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.217574    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 01:21:20.245539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.234448    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.313633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.306623    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.313633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.308261    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323377    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323409    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323481    4576 cpu_manager.go:414] "containerMap: removing container" podUID="93f06dd1-fc1b-498a-8658-5639c6bd0951" containerName="payload-pre-processing"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323490    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f06dd1-fc1b-498a-8658-5639c6bd0951" containerName="payload-pre-processing"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323506    4576 cpu_manager.go:414] "containerMap: removing container" podUID="07a9b46f-d980-4465-9dcf-158a3b2d9de0" containerName="payload-processing"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323513    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a9b46f-d980-4465-9dcf-158a3b2d9de0" containerName="payload-processing"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323692    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f06dd1-fc1b-498a-8658-5639c6bd0951" containerName="payload-pre-processing"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323710    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a9b46f-d980-4465-9dcf-158a3b2d9de0" containerName="payload-processing"
Oct 30 01:21:20.326276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.323727    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" containerName="main"
Oct 30 01:21:20.393597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.393473    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:20.394866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.394839    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"]
Oct 30 01:21:20.395064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.395039    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.408282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.407335    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.435160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.435120    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 01:21:20.436209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.436191    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.453338    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.453776    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.454665    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tls-certs\") pod \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") "
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.454749    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-model-cache\") pod \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") "
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.454830    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-kube-api-access-9hjf5\" (UniqueName: \"kubernetes.io/projected/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-kube-api-access-9hjf5\") pod \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") "
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.454933    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-home\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-home\") pod \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") "
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.454975    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-dshm\") pod \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") "
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.455092    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tmp-dir\") pod \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\" (UID: \"ddbd16dd-6a19-42e2-9406-fc4d963a53c2\") "
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.455415    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.455741    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.455800    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krts8\" (UniqueName: \"kubernetes.io/projected/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-kube-api-access-krts8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.462060    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-model-cache" pod "ddbd16dd-6a19-42e2-9406-fc4d963a53c2" (UID: "ddbd16dd-6a19-42e2-9406-fc4d963a53c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.462291    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-home" pod "ddbd16dd-6a19-42e2-9406-fc4d963a53c2" (UID: "ddbd16dd-6a19-42e2-9406-fc4d963a53c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.462899    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tmp-dir" pod "ddbd16dd-6a19-42e2-9406-fc4d963a53c2" (UID: "ddbd16dd-6a19-42e2-9406-fc4d963a53c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.479900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.463492    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:21:20.521486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.520759    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:21:20.521486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.521070    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.568156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.568118    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.568443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.568419    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.568687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.568668    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krts8\" (UniqueName: \"kubernetes.io/projected/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-kube-api-access-krts8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.569000 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.568977    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.573255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.569452    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.573445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.573427    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.573547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.573535    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.583301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.577486    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.584525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.584412    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tls-certs" pod "ddbd16dd-6a19-42e2-9406-fc4d963a53c2" (UID: "ddbd16dd-6a19-42e2-9406-fc4d963a53c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:20.584525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.584489    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-dshm" pod "ddbd16dd-6a19-42e2-9406-fc4d963a53c2" (UID: "ddbd16dd-6a19-42e2-9406-fc4d963a53c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.649818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.608339    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:20.652205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.650436    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:20.652205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.650496    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:20.655749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.654777    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.656855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.656805    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-kube-api-access-9hjf5" pod "ddbd16dd-6a19-42e2-9406-fc4d963a53c2" (UID: "ddbd16dd-6a19-42e2-9406-fc4d963a53c2"). InnerVolumeSpecName "kube-api-access-9hjf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:20.658081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.657977    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:21:20.658081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.658021    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 01:21:20.662940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.662200    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 01:21:20.701464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.689123    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hjf5\" (UniqueName: \"kubernetes.io/projected/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-kube-api-access-9hjf5\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.701464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.689160    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.701464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.689176    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbd16dd-6a19-42e2-9406-fc4d963a53c2-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.703559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.703448    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 01:21:20.703559 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.703504    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 01:21:20.748371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.748333    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"]
Oct 30 01:21:20.752266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.752136    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krts8\" (UniqueName: \"kubernetes.io/projected/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-kube-api-access-krts8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.834465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.824966    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08beaa65-bs5rx"]
Oct 30 01:21:20.914659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.914330    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:20.971480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:20.971438    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:21.060252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.046434    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 01:21:21.072438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.071346    4576 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Oct 30 01:21:21.094682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.094643    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:21.131267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.122587    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 01:21:21.131267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.122724    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:21.131267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.125819    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:21.143032 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.142983    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:21:21.248447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.248407    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:21.433277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.248609    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"
Oct 30 01:21:21.433277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.248639    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded"
Oct 30 01:21:21.503341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.370700    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:21.503341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.400369    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:21.503341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.497579    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:21.503341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.497926    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:21.503341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.498284    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:21:21.508249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.508189    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 01:21:21.543881 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.543663    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 01:21:21.545328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.545295    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 01:21:21.610263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.554017    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 01:21:21.610263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.602377    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:21.666651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.666607    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 01:21:21.678733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.678699    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:21.679028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.678989    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:21.734977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.734937    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:21.755761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.755670    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 01:21:21.756067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.756019    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 01:21:21.810961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.810923    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:21.905179 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.905153    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:21:22.009563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:21.994097    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:21:22.045008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.040356    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:21:22.097122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.097089    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 01:21:22.206652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.206616    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d77e55-3c6b-4ec9-a01b-8c5df878245d" path="/var/lib/kubelet/pods/01d77e55-3c6b-4ec9-a01b-8c5df878245d/volumes"
Oct 30 01:21:22.265302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.261135    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:22.265302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.261214    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 01:21:22.265302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.261302    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 01:21:22.265302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.261335    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 01:21:22.265302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.261476    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:21:22.265302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.261534    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 01:21:22.318903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.316957    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:22.538321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.523750    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:21:22.624450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.624398    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"]
Oct 30 01:21:22.640877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.640834    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 01:21:22.641073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.640897    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 01:21:22.641073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.640934    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ad00a0/auth-test-model-ad00a0-kserve-548b8d8dc7-zwtsn"]
Oct 30 01:21:22.714285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.710729    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"]
Oct 30 01:21:22.723192 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:22.723155    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f250b27_b9be_41e8_8950_ab4c8a0dee9e.slice/crio-9a47ed8f029e8f434bf9920535b71540d53385e88e15d9e578b2cab3ac47301f WatchSource:0}: Error finding container 9a47ed8f029e8f434bf9920535b71540d53385e88e15d9e578b2cab3ac47301f: Status 404 returned error can't find the container with id 9a47ed8f029e8f434bf9920535b71540d53385e88e15d9e578b2cab3ac47301f
Oct 30 01:21:22.745528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.745498    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:22.748852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.748831    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:22.757627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.753816    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:21:22.876278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.873541    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:21:22.956025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.955989    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-data" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:22.956217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.955651    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-data\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.956217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.956136    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-istio-token\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-istio-token\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.956217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.956205    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-socket\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.956721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.956695    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-certs\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.956823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.956740    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13bc6efc-6a25-4023-be95-ffb996da747c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bc6efc-6a25-4023-be95-ffb996da747c-istio-podinfo\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.956823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.956774    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-socket" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:22.956823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.956796    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-credential-socket\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.957004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.956865    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-envoy\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.957004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.956995    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-certs" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:22.957113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.957024    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13bc6efc-6a25-4023-be95-ffb996da747c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bc6efc-6a25-4023-be95-ffb996da747c-istiod-ca-cert\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.957113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.957062    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-kube-api-access-rb6hk\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-kube-api-access-rb6hk\") pod \"13bc6efc-6a25-4023-be95-ffb996da747c\" (UID: \"13bc6efc-6a25-4023-be95-ffb996da747c\") "
Oct 30 01:21:22.957673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.957646    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-credential-socket" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:22.964277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.957872    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:22.964277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.959752    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:22.964277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.959772    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:22.964277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.959786    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:22.964277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.959594    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bc6efc-6a25-4023-be95-ffb996da747c-istiod-ca-cert" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:22.969537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.969509    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-envoy" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:22.971372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.971342    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-istio-token" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:22.971579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.971491    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13bc6efc-6a25-4023-be95-ffb996da747c-istio-podinfo" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:22.971671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:22.971583    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-kube-api-access-rb6hk" pod "13bc6efc-6a25-4023-be95-ffb996da747c" (UID: "13bc6efc-6a25-4023-be95-ffb996da747c"). InnerVolumeSpecName "kube-api-access-rb6hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:23.061474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.061392    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13bc6efc-6a25-4023-be95-ffb996da747c-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:23.061474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.061427    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13bc6efc-6a25-4023-be95-ffb996da747c-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:23.061474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.061439    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6hk\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-kube-api-access-rb6hk\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:23.061474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.061449    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13bc6efc-6a25-4023-be95-ffb996da747c-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:23.061474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.061460    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13bc6efc-6a25-4023-be95-ffb996da747c-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:23.369552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.368704    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg" event={"ID":"3f250b27-b9be-41e8-8950-ab4c8a0dee9e","Type":"ContainerStarted","Data":"8fa473d12916ddb61b3202636a56f3956ee92f04b3d9b6b5a0c9bcfd6a472339"}
Oct 30 01:21:23.369552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.368783    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg" event={"ID":"3f250b27-b9be-41e8-8950-ab4c8a0dee9e","Type":"ContainerStarted","Data":"9a47ed8f029e8f434bf9920535b71540d53385e88e15d9e578b2cab3ac47301f"}
Oct 30 01:21:23.371731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.371693    4576 generic.go:356] "Generic (PLEG): container finished" podID="13bc6efc-6a25-4023-be95-ffb996da747c" containerID="ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd" exitCode=0
Oct 30 01:21:23.371886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.371740    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"
Oct 30 01:21:23.371886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.371796    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" event={"ID":"13bc6efc-6a25-4023-be95-ffb996da747c","Type":"ContainerDied","Data":"ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd"}
Oct 30 01:21:23.371886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.371858    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77" event={"ID":"13bc6efc-6a25-4023-be95-ffb996da747c","Type":"ContainerDied","Data":"2564d8a5336775d67a7b57bdf54dd838a5c6e183be0da89f1cca119a7706f718"}
Oct 30 01:21:23.371886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.371883    4576 scope.go:122] "RemoveContainer" containerID="ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd"
Oct 30 01:21:23.401992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.401962    4576 scope.go:122] "RemoveContainer" containerID="ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd"
Oct 30 01:21:23.402482 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:21:23.402451    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd\": container with ID starting with ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd not found: ID does not exist" containerID="ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd"
Oct 30 01:21:23.402606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.402495    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd"} err="failed to get container status \"ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd\": rpc error: code = NotFound desc = could not find container \"ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd\": container with ID starting with ad876fa6c1765a11473aea3e4e86020b35023584b831a2c35c8f7929737612cd not found: ID does not exist"
Oct 30 01:21:23.431075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.431018    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg" podStartSLOduration=6.43100435 podStartE2EDuration="6.43100435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:23.426856727 +0000 UTC m=+2563.498180564" watchObservedRunningTime="2026-10-30 01:21:23.43100435 +0000 UTC m=+2563.502328200"
Oct 30 01:21:23.474934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.474902    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"]
Oct 30 01:21:23.485920 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.485894    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-283eb6-openshift-default-bc7b7cc95-s7v77"]
Oct 30 01:21:23.644872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.644793    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:23.645365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.645348    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:23.645990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:23.645968    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:21:24.110462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:24.110431    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" path="/var/lib/kubelet/pods/13bc6efc-6a25-4023-be95-ffb996da747c/volumes"
Oct 30 01:21:24.111034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:24.111018    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbd16dd-6a19-42e2-9406-fc4d963a53c2" path="/var/lib/kubelet/pods/ddbd16dd-6a19-42e2-9406-fc4d963a53c2/volumes"
Oct 30 01:21:24.316767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:24.316728    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:24.382270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:24.382220    4576 generic.go:356] "Generic (PLEG): container finished" podID="3f250b27-b9be-41e8-8950-ab4c8a0dee9e" containerID="8fa473d12916ddb61b3202636a56f3956ee92f04b3d9b6b5a0c9bcfd6a472339" exitCode=0
Oct 30 01:21:24.382456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:24.382272    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg" event={"ID":"3f250b27-b9be-41e8-8950-ab4c8a0dee9e","Type":"ContainerDied","Data":"8fa473d12916ddb61b3202636a56f3956ee92f04b3d9b6b5a0c9bcfd6a472339"}
Oct 30 01:21:25.028265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.028213    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"]
Oct 30 01:21:25.041749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.041717    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-464b3050-nt84v"]
Oct 30 01:21:25.352837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.352751    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:25.495026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.494992    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:25.565329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.565301    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:25.611084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.610998    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-kube-api-access-krts8\" (UniqueName: \"kubernetes.io/projected/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-kube-api-access-krts8\") pod \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") "
Oct 30 01:21:25.618686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.617719    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-xks-ca\") pod \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") "
Oct 30 01:21:25.618686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.618084    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-openshift-ca\") pod \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\" (UID: \"3f250b27-b9be-41e8-8950-ab4c8a0dee9e\") "
Oct 30 01:21:25.619955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.619787    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-openshift-ca" pod "3f250b27-b9be-41e8-8950-ab4c8a0dee9e" (UID: "3f250b27-b9be-41e8-8950-ab4c8a0dee9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:25.631400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.630404    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-kube-api-access-krts8" pod "3f250b27-b9be-41e8-8950-ab4c8a0dee9e" (UID: "3f250b27-b9be-41e8-8950-ab4c8a0dee9e"). InnerVolumeSpecName "kube-api-access-krts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:25.635726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.634705    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:25.635887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.635829    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-xks-ca" pod "3f250b27-b9be-41e8-8950-ab4c8a0dee9e" (UID: "3f250b27-b9be-41e8-8950-ab4c8a0dee9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:25.721449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.721390    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:25.721449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.721443    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:25.721449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.721457    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krts8\" (UniqueName: \"kubernetes.io/projected/3f250b27-b9be-41e8-8950-ab4c8a0dee9e-kube-api-access-krts8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:21:25.751117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:25.751071    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:26.127269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:26.126834    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb" path="/var/lib/kubelet/pods/a72f61cc-5b6a-4ef2-91d6-e83e73ff1bcb/volumes"
Oct 30 01:21:26.401965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:26.401873    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"
Oct 30 01:21:26.401965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:26.401904    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg" event={"ID":"3f250b27-b9be-41e8-8950-ab4c8a0dee9e","Type":"ContainerDied","Data":"9a47ed8f029e8f434bf9920535b71540d53385e88e15d9e578b2cab3ac47301f"}
Oct 30 01:21:26.401965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:26.401955    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a47ed8f029e8f434bf9920535b71540d53385e88e15d9e578b2cab3ac47301f"
Oct 30 01:21:26.667776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:26.667688    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:27.295375 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.295344    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"]
Oct 30 01:21:27.296074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.296058    4576 cpu_manager.go:414] "containerMap: removing container" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy"
Oct 30 01:21:27.296125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.296075    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy"
Oct 30 01:21:27.296125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.296092    4576 cpu_manager.go:414] "containerMap: removing container" podUID="3f250b27-b9be-41e8-8950-ab4c8a0dee9e" containerName="revoke-keys"
Oct 30 01:21:27.296125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.296097    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f250b27-b9be-41e8-8950-ab4c8a0dee9e" containerName="revoke-keys"
Oct 30 01:21:27.296294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.296211    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="13bc6efc-6a25-4023-be95-ffb996da747c" containerName="istio-proxy"
Oct 30 01:21:27.296294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.296224    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f250b27-b9be-41e8-8950-ab4c8a0dee9e" containerName="revoke-keys"
Oct 30 01:21:27.300134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.300117    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.346570 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.346529    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"]
Oct 30 01:21:27.361126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361091    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.361290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361131    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.361290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361244    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.361290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361283    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.361484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361423    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd73bed-f9a0-436c-aead-a7aa427c9087-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.361549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361493    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.361602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361578    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwd8r\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-kube-api-access-rwd8r\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.361781 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361765    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.361854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.361820    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.462994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.462951    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwd8r\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-kube-api-access-rwd8r\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463146    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463197    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463290    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463315    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463363    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463396    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463485    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd73bed-f9a0-436c-aead-a7aa427c9087-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463518    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463757    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-credential-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.463967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463948    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-socket\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.464075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463888    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-certs\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.464075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.463803    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-data\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.464279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.464257    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd73bed-f9a0-436c-aead-a7aa427c9087-istiod-ca-cert\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.465860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.465836    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-envoy\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.466274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.466219    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-podinfo\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.484491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.484466    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-token\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.484618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.484600    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwd8r\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-kube-api-access-rwd8r\") pod \"e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") " pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.617524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.617443    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:27.940630 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:27.940600    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd73bed_f9a0_436c_aead_a7aa427c9087.slice/crio-b2b4be964736d85746c1af0ef2a5c2e0ea65b34d7ec5700ec8402cc5b77ddc6b WatchSource:0}: Error finding container b2b4be964736d85746c1af0ef2a5c2e0ea65b34d7ec5700ec8402cc5b77ddc6b: Status 404 returned error can't find the container with id b2b4be964736d85746c1af0ef2a5c2e0ea65b34d7ec5700ec8402cc5b77ddc6b
Oct 30 01:21:27.941178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.941156    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"]
Oct 30 01:21:27.944016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.943979    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:27.944137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.944062    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:27.944137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:27.944112    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:28.086206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.086157    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:28.212006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.211925    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:28.242180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.242145    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"]
Oct 30 01:21:28.246809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.246784    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:28.263393 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.263362    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96bd13\""
Oct 30 01:21:28.263539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.263510    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96bd13-dockercfg-bhskd\""
Oct 30 01:21:28.268807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.268747    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"]
Oct 30 01:21:28.392256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.388442    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfh2s\" (UniqueName: \"kubernetes.io/projected/5fd49eaf-15c9-4507-9e2b-caa4147544af-kube-api-access-kfh2s\") pod \"payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d\" (UID: \"5fd49eaf-15c9-4507-9e2b-caa4147544af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:28.392256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.388603    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd49eaf-15c9-4507-9e2b-caa4147544af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d\" (UID: \"5fd49eaf-15c9-4507-9e2b-caa4147544af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:28.566414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.561409    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd49eaf-15c9-4507-9e2b-caa4147544af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d\" (UID: \"5fd49eaf-15c9-4507-9e2b-caa4147544af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:28.566414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.561791    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfh2s\" (UniqueName: \"kubernetes.io/projected/5fd49eaf-15c9-4507-9e2b-caa4147544af-kube-api-access-kfh2s\") pod \"payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d\" (UID: \"5fd49eaf-15c9-4507-9e2b-caa4147544af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:28.566414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.563062    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd49eaf-15c9-4507-9e2b-caa4147544af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d\" (UID: \"5fd49eaf-15c9-4507-9e2b-caa4147544af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:28.634389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.623611    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfh2s\" (UniqueName: \"kubernetes.io/projected/5fd49eaf-15c9-4507-9e2b-caa4147544af-kube-api-access-kfh2s\") pod \"payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d\" (UID: \"5fd49eaf-15c9-4507-9e2b-caa4147544af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:28.634389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.624887    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:28.634389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.628388    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" event={"ID":"0bd73bed-f9a0-436c-aead-a7aa427c9087","Type":"ContainerStarted","Data":"0227d4a793de63e747ec4def1f82f9cf81e57f9685e92b4a4ffcae4d44c63958"}
Oct 30 01:21:28.634389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.628485    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" event={"ID":"0bd73bed-f9a0-436c-aead-a7aa427c9087","Type":"ContainerStarted","Data":"b2b4be964736d85746c1af0ef2a5c2e0ea65b34d7ec5700ec8402cc5b77ddc6b"}
Oct 30 01:21:28.681493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.680408    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" podStartSLOduration=1.680387508 podStartE2EDuration="1.680387508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:28.679698015 +0000 UTC m=+2568.751021863" watchObservedRunningTime="2026-10-30 01:21:28.680387508 +0000 UTC m=+2568.751711343"
Oct 30 01:21:28.772856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.772818    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:28.863208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.863128    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:28.881035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:28.880906    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:29.189188 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:29.189160    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"]
Oct 30 01:21:29.618357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:29.618327    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:29.620386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:29.620356    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:21:29.620654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:29.620594    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 01:21:29.638964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:29.638921    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d" event={"ID":"5fd49eaf-15c9-4507-9e2b-caa4147544af","Type":"ContainerStarted","Data":"ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39"}
Oct 30 01:21:29.639137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:29.638985    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d" event={"ID":"5fd49eaf-15c9-4507-9e2b-caa4147544af","Type":"ContainerStarted","Data":"c3e3f9d030353f2580662820b03a7ea3f00b60ba52668f1b3727172d93046dff"}
Oct 30 01:21:29.695661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:29.695584    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d" podStartSLOduration=1.695563852 podStartE2EDuration="1.695563852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:29.68182165 +0000 UTC m=+2569.753145486" watchObservedRunningTime="2026-10-30 01:21:29.695563852 +0000 UTC m=+2569.766887687"
Oct 30 01:21:30.618107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:30.618074    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:21:30.618379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:30.618128    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 01:21:30.651613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:30.651574    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:31.099008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:31.098957    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:31.617784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:31.617742    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:21:31.617966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:31.617800    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 01:21:32.618871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:32.618831    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:21:32.619399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:32.618893    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 01:21:32.767203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:32.767162    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:33.618067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:33.618032    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:21:33.618262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:33.618094    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 01:21:34.400909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:34.400872    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:34.618586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:34.618546    4576 patch_prober.go:29] interesting pod/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 01:21:34.618754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:34.618607    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 01:21:35.029759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:35.029712    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"]
Oct 30 01:21:35.043433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:35.043399    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0afe36-swtj6"]
Oct 30 01:21:35.623437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:35.623402    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:35.623841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:35.623794    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:35.624621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:35.624603    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:21:36.111419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.111372    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc37556a-5835-4b59-9e74-a4034961304a" path="/var/lib/kubelet/pods/cc37556a-5835-4b59-9e74-a4034961304a/volumes"
Oct 30 01:21:36.203687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.203653    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:36.378071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.377992    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"]
Oct 30 01:21:36.384591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.384565    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.403836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.403809    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"]
Oct 30 01:21:36.412005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.411975    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-954bf6\"/\"default-dockercfg-xlrll\""
Oct 30 01:21:36.416759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.416735    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-954bf6\"/\"kube-root-ca.crt\""
Oct 30 01:21:36.416759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.416751    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-954bf6\"/\"auth-test-model-954bf6-kserve-self-signed-certs\""
Oct 30 01:21:36.416898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.416770    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-954bf6\"/\"openshift-service-ca.crt\""
Oct 30 01:21:36.457311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.457258    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:36.499133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.499096    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc83f61-3437-423b-b0e6-096c8f017880-tls-certs\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.499350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.499138    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-tmp-dir\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.499414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.499353    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-model-cache\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.499463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.499424    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-home\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.499561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.499540    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsm9t\" (UniqueName: \"kubernetes.io/projected/fdc83f61-3437-423b-b0e6-096c8f017880-kube-api-access-rsm9t\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.499613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.499575    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-dshm\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.602339 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.602302    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc83f61-3437-423b-b0e6-096c8f017880-tls-certs\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.603917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.603891    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-tmp-dir\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.605844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.605809    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-model-cache\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.605958 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.605883    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-home\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.606031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.605989    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsm9t\" (UniqueName: \"kubernetes.io/projected/fdc83f61-3437-423b-b0e6-096c8f017880-kube-api-access-rsm9t\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.606031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.606026    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-dshm\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.606855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.606834    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-model-cache\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.606998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.604618    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-tmp-dir\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.607327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.607307    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-home\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.609708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.609661    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-dshm\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.644548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.644462    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc83f61-3437-423b-b0e6-096c8f017880-tls-certs\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.647939 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.647867    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsm9t\" (UniqueName: \"kubernetes.io/projected/fdc83f61-3437-423b-b0e6-096c8f017880-kube-api-access-rsm9t\") pod \"auth-test-model-954bf6-kserve-795b685bd9-47wqv\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") " pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:36.706693 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:36.706637    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:37.031151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:37.031106    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"]
Oct 30 01:21:37.037298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:37.037264    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0179dbb6-2mdhw"]
Oct 30 01:21:37.130565 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:37.130529    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdc83f61_3437_423b_b0e6_096c8f017880.slice/crio-b20381b50de3e79512b7fa5e2201be0e0a8f467f25715c0079be43b0632bc7e7 WatchSource:0}: Error finding container b20381b50de3e79512b7fa5e2201be0e0a8f467f25715c0079be43b0632bc7e7: Status 404 returned error can't find the container with id b20381b50de3e79512b7fa5e2201be0e0a8f467f25715c0079be43b0632bc7e7
Oct 30 01:21:37.130897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:37.130876    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"]
Oct 30 01:21:37.773966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:37.773930    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv" event={"ID":"fdc83f61-3437-423b-b0e6-096c8f017880","Type":"ContainerStarted","Data":"113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5"}
Oct 30 01:21:37.774423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:37.773976    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv" event={"ID":"fdc83f61-3437-423b-b0e6-096c8f017880","Type":"ContainerStarted","Data":"b20381b50de3e79512b7fa5e2201be0e0a8f467f25715c0079be43b0632bc7e7"}
Oct 30 01:21:37.817893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:37.817841    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv" podStartSLOduration=1.8178224589999998 podStartE2EDuration="1.817822459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:37.813393073 +0000 UTC m=+2577.884716908" watchObservedRunningTime="2026-10-30 01:21:37.817822459 +0000 UTC m=+2577.889146293"
Oct 30 01:21:38.062705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:38.062611    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:38.112297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:38.112257    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e320070-870c-4d31-9a84-f4981f106f1d" path="/var/lib/kubelet/pods/8e320070-870c-4d31-9a84-f4981f106f1d/volumes"
Oct 30 01:21:38.715337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:38.715304    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:39.100361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:39.100269    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:21:40.190401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:40.190362    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:40.305256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:40.299431    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:41.454865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.454056    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"]
Oct 30 01:21:41.466258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.463821    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.485774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.485254    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"]
Oct 30 01:21:41.606844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.606811    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.607018 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.606966    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.607018 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.606999    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.607141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.607076    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.607141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.607116    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.607220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.607157    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.607360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.607334    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.607478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.607451    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrx5\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-kube-api-access-xdrx5\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.607619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.607511    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.662770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.662739    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:21:41.709294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709198    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709334    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709383    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709449    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709470    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709514    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709582    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709634    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrx5\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-kube-api-access-xdrx5\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709669    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709680    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-certs\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.709928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.709750    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.710161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.710143    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-data\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.710380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.710360    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-credential-socket\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.710554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.710532    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istiod-ca-cert\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.712324 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.712300    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-envoy\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.712974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.712941    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-podinfo\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.719553 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.719441    4576 scope.go:122] "RemoveContainer" containerID="7cad2c55e89c2ff50bb1df84275aa6c93701df7b13f45506d6d72b9ff59996dd"
Oct 30 01:21:41.734703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.734676    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-token\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.737075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.737025    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrx5\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-kube-api-access-xdrx5\") pod \"e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") " pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:41.767558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.767533    4576 scope.go:122] "RemoveContainer" containerID="522833fb268db7a739d730a9fd335567d931c2bc2baa67fb11e9d7a05703c116"
Oct 30 01:21:41.802735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:41.802707    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:42.146246 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:42.146208    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"]
Oct 30 01:21:42.147152 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:21:42.147129    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8c9b6e_320d_4308_b9c2_01e9489a99fb.slice/crio-018969fdcfe8ac6d57aa558df75df9ff062b7355839fc45375f88beacac0cae5 WatchSource:0}: Error finding container 018969fdcfe8ac6d57aa558df75df9ff062b7355839fc45375f88beacac0cae5: Status 404 returned error can't find the container with id 018969fdcfe8ac6d57aa558df75df9ff062b7355839fc45375f88beacac0cae5
Oct 30 01:21:42.150759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:42.150726    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:42.150866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:42.150795    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:42.150866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:42.150825    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:21:42.704263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:42.701623    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:42.865033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:42.865000    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" event={"ID":"ae8c9b6e-320d-4308-b9c2-01e9489a99fb","Type":"ContainerStarted","Data":"e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219"}
Oct 30 01:21:42.865246 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:42.865059    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" event={"ID":"ae8c9b6e-320d-4308-b9c2-01e9489a99fb","Type":"ContainerStarted","Data":"018969fdcfe8ac6d57aa558df75df9ff062b7355839fc45375f88beacac0cae5"}
Oct 30 01:21:42.940719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:42.940671    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" podStartSLOduration=1.940658081 podStartE2EDuration="1.940658081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:42.923378252 +0000 UTC m=+2582.994702092" watchObservedRunningTime="2026-10-30 01:21:42.940658081 +0000 UTC m=+2583.011981916"
Oct 30 01:21:43.742517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:43.742482    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:43.803537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:43.803502    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:43.805554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:43.805523    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:21:43.805693 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:43.805577    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 01:21:44.395259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:44.394414    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:44.536583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:44.536548    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:44.803825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:44.803787    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:21:44.804280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:44.803868    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 01:21:45.803925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:45.803892    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:21:45.804395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:45.803976    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 01:21:46.093047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:46.092944    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:46.715617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:46.715584    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:46.728567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:46.728542    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:46.803903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:46.803867    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:21:46.804100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:46.803948    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 01:21:46.936160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:46.936135    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:21:47.030327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:47.030222    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"]
Oct 30 01:21:47.034685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:47.034658    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8866816-bxgmj"]
Oct 30 01:21:47.803620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:47.803580    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 01:21:47.803876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:47.803637    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 01:21:47.913892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:47.913833    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:48.110666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:48.110593    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b90aab1-a761-420c-a30e-b503402360b7" path="/var/lib/kubelet/pods/7b90aab1-a761-420c-a30e-b503402360b7/volumes"
Oct 30 01:21:48.386154 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:21:48.386065    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:21:48.809025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:48.808994    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:48.809551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:48.809534    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:48.810282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:48.810256    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:21:49.480946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:49.480907    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:51.014465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:51.014415    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:52.796248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:52.796202    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:54.061140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:54.061104    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:54.169666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:54.169453    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:55.703543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:55.703499    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:57.802315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:57.802278    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:57.921363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:57.921326    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:21:59.675100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:21:59.675065    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:01.140348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:01.140307    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:02.587017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:02.586985    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:04.278057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:04.278027    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:05.744037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:05.743987    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:07.325861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:07.325831    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:09.108750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:09.108698    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:10.346156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:10.346118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:10.472631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:10.472592    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:11.071680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.071641    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"]
Oct 30 01:22:11.076357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.076334    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.092291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.092264    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"]
Oct 30 01:22:11.100389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.100367    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:11.100571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.100518    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:22:11.137128 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.137086    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8422\" (UniqueName: \"kubernetes.io/projected/66c7e342-20f8-4827-9d14-9b4a01d65e52-kube-api-access-q8422\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.137537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.137502    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.137693 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.137591    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.239760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.239722    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.239959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.239785    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.240034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.239961    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8422\" (UniqueName: \"kubernetes.io/projected/66c7e342-20f8-4827-9d14-9b4a01d65e52-kube-api-access-q8422\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.240527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.240504    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.242440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.242418    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.260983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.260954    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8422\" (UniqueName: \"kubernetes.io/projected/66c7e342-20f8-4827-9d14-9b4a01d65e52-kube-api-access-q8422\") pod \"maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.395499 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.395425    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:11.512157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.512125    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:11.703051 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:22:11.703020    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c7e342_20f8_4827_9d14_9b4a01d65e52.slice/crio-d65711ae79e063785094e543ea91ef48147c343cb4e14aae8cd98bd5d7685402 WatchSource:0}: Error finding container d65711ae79e063785094e543ea91ef48147c343cb4e14aae8cd98bd5d7685402: Status 404 returned error can't find the container with id d65711ae79e063785094e543ea91ef48147c343cb4e14aae8cd98bd5d7685402
Oct 30 01:22:11.703222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:11.703158    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"]
Oct 30 01:22:12.169143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:12.169097    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk" event={"ID":"66c7e342-20f8-4827-9d14-9b4a01d65e52","Type":"ContainerStarted","Data":"c67c657d75d82332215910ff7ebe14678dd46ded20f5816bf7e712bfb75ca72f"}
Oct 30 01:22:12.169367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:12.169162    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk" event={"ID":"66c7e342-20f8-4827-9d14-9b4a01d65e52","Type":"ContainerStarted","Data":"d65711ae79e063785094e543ea91ef48147c343cb4e14aae8cd98bd5d7685402"}
Oct 30 01:22:12.213412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:12.213352    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk" podStartSLOduration=1.213338448 podStartE2EDuration="1.213338448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:12.211404585 +0000 UTC m=+2612.282728422" watchObservedRunningTime="2026-10-30 01:22:12.213338448 +0000 UTC m=+2612.284662282"
Oct 30 01:22:13.179289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:13.179252    4576 generic.go:356] "Generic (PLEG): container finished" podID="66c7e342-20f8-4827-9d14-9b4a01d65e52" containerID="c67c657d75d82332215910ff7ebe14678dd46ded20f5816bf7e712bfb75ca72f" exitCode=0
Oct 30 01:22:13.179787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:13.179344    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk" event={"ID":"66c7e342-20f8-4827-9d14-9b4a01d65e52","Type":"ContainerDied","Data":"c67c657d75d82332215910ff7ebe14678dd46ded20f5816bf7e712bfb75ca72f"}
Oct 30 01:22:13.350672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:13.350638    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:14.347351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.347321    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:14.496761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.496676    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-xks-ca\") pod \"66c7e342-20f8-4827-9d14-9b4a01d65e52\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") "
Oct 30 01:22:14.496960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.496946    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c7e342-20f8-4827-9d14-9b4a01d65e52-kube-api-access-q8422\" (UniqueName: \"kubernetes.io/projected/66c7e342-20f8-4827-9d14-9b4a01d65e52-kube-api-access-q8422\") pod \"66c7e342-20f8-4827-9d14-9b4a01d65e52\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") "
Oct 30 01:22:14.497028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.496994    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-openshift-ca\") pod \"66c7e342-20f8-4827-9d14-9b4a01d65e52\" (UID: \"66c7e342-20f8-4827-9d14-9b4a01d65e52\") "
Oct 30 01:22:14.497453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.497420    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-openshift-ca" pod "66c7e342-20f8-4827-9d14-9b4a01d65e52" (UID: "66c7e342-20f8-4827-9d14-9b4a01d65e52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:14.497609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.497535    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:14.498975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.498948    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-xks-ca" pod "66c7e342-20f8-4827-9d14-9b4a01d65e52" (UID: "66c7e342-20f8-4827-9d14-9b4a01d65e52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:14.499253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.499212    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c7e342-20f8-4827-9d14-9b4a01d65e52-kube-api-access-q8422" pod "66c7e342-20f8-4827-9d14-9b4a01d65e52" (UID: "66c7e342-20f8-4827-9d14-9b4a01d65e52"). InnerVolumeSpecName "kube-api-access-q8422". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:14.598994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.598953    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c7e342-20f8-4827-9d14-9b4a01d65e52-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:14.598994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:14.598986    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8422\" (UniqueName: \"kubernetes.io/projected/66c7e342-20f8-4827-9d14-9b4a01d65e52-kube-api-access-q8422\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:15.020466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:15.020431    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:15.139512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:15.139474    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:15.234477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:15.234383    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk" event={"ID":"66c7e342-20f8-4827-9d14-9b4a01d65e52","Type":"ContainerDied","Data":"d65711ae79e063785094e543ea91ef48147c343cb4e14aae8cd98bd5d7685402"}
Oct 30 01:22:15.234694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:15.234500    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d65711ae79e063785094e543ea91ef48147c343cb4e14aae8cd98bd5d7685402"
Oct 30 01:22:15.234694 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:15.234569    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"
Oct 30 01:22:15.248098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:15.248058    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:16.449625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:16.449591    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:17.227565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:17.227527    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:17.384868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:17.384832    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:17.400223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:17.400193    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"]
Oct 30 01:22:17.413643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:17.413616    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-43d152-67ggg"]
Oct 30 01:22:17.523190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:17.523112    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:17.885299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:17.885211    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:18.121106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:18.121066    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f250b27-b9be-41e8-8950-ab4c8a0dee9e" path="/var/lib/kubelet/pods/3f250b27-b9be-41e8-8950-ab4c8a0dee9e/volumes"
Oct 30 01:22:18.384527 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:22:18.384461    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:22:19.320256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:19.311945    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:22.696451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:22.696419    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:23.762371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:23.762330    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:23.878075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:23.878032    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:24.027732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.027641    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"]
Oct 30 01:22:24.028305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.028249    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy" containerID="cri-o://70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9" gracePeriod=30
Oct 30 01:22:24.075957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.075924    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:24.524785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.524753    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:24.659035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.659005    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"]
Oct 30 01:22:24.659745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.659729    4576 cpu_manager.go:414] "containerMap: removing container" podUID="66c7e342-20f8-4827-9d14-9b4a01d65e52" containerName="revoke-keys"
Oct 30 01:22:24.659803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.659747    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c7e342-20f8-4827-9d14-9b4a01d65e52" containerName="revoke-keys"
Oct 30 01:22:24.659859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.659850    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c7e342-20f8-4827-9d14-9b4a01d65e52" containerName="revoke-keys"
Oct 30 01:22:24.664721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.664698    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.685900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.685296    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"]
Oct 30 01:22:24.692740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.692715    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:24.692869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.692774    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:22:24.697040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.697010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:24.697221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.697078    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.697221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.697156    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.697466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.697447    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stpjd\" (UniqueName: \"kubernetes.io/projected/46eff990-edaf-4db0-bf11-ecf53146ed0d-kube-api-access-stpjd\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.798727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.798643    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stpjd\" (UniqueName: \"kubernetes.io/projected/46eff990-edaf-4db0-bf11-ecf53146ed0d-kube-api-access-stpjd\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.799139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.798823    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.799139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.798871    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.799644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.799622    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.801843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.801817    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.832890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.832827    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stpjd\" (UniqueName: \"kubernetes.io/projected/46eff990-edaf-4db0-bf11-ecf53146ed0d-kube-api-access-stpjd\") pod \"maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:24.989826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:24.989793    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:25.296868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:25.296836    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"]
Oct 30 01:22:25.297916 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:22:25.297889    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46eff990_edaf_4db0_bf11_ecf53146ed0d.slice/crio-bdf8fe105bf040a3ce7655a4e3deeb3ebaba39cdd3df09e7739bcdbaed5c35c9 WatchSource:0}: Error finding container bdf8fe105bf040a3ce7655a4e3deeb3ebaba39cdd3df09e7739bcdbaed5c35c9: Status 404 returned error can't find the container with id bdf8fe105bf040a3ce7655a4e3deeb3ebaba39cdd3df09e7739bcdbaed5c35c9
Oct 30 01:22:25.382002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:25.381970    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl" event={"ID":"46eff990-edaf-4db0-bf11-ecf53146ed0d","Type":"ContainerStarted","Data":"bdf8fe105bf040a3ce7655a4e3deeb3ebaba39cdd3df09e7739bcdbaed5c35c9"}
Oct 30 01:22:26.391308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:26.391262    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl" event={"ID":"46eff990-edaf-4db0-bf11-ecf53146ed0d","Type":"ContainerDied","Data":"43ab6524556fa702245f24ce08d745efdbb08694a22c391be287e38c0ec596a5"}
Oct 30 01:22:26.391944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:26.391225    4576 generic.go:356] "Generic (PLEG): container finished" podID="46eff990-edaf-4db0-bf11-ecf53146ed0d" containerID="43ab6524556fa702245f24ce08d745efdbb08694a22c391be287e38c0ec596a5" exitCode=0
Oct 30 01:22:26.776141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:26.776109    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:27.585925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.585901    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:27.646307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.646272    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eff990-edaf-4db0-bf11-ecf53146ed0d-kube-api-access-stpjd\" (UniqueName: \"kubernetes.io/projected/46eff990-edaf-4db0-bf11-ecf53146ed0d-kube-api-access-stpjd\") pod \"46eff990-edaf-4db0-bf11-ecf53146ed0d\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") "
Oct 30 01:22:27.646501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.646414    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-openshift-ca\") pod \"46eff990-edaf-4db0-bf11-ecf53146ed0d\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") "
Oct 30 01:22:27.646501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.646488    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-xks-ca\") pod \"46eff990-edaf-4db0-bf11-ecf53146ed0d\" (UID: \"46eff990-edaf-4db0-bf11-ecf53146ed0d\") "
Oct 30 01:22:27.646960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.646931    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-openshift-ca" pod "46eff990-edaf-4db0-bf11-ecf53146ed0d" (UID: "46eff990-edaf-4db0-bf11-ecf53146ed0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:27.647415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.647383    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.648845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.648814    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eff990-edaf-4db0-bf11-ecf53146ed0d-kube-api-access-stpjd" pod "46eff990-edaf-4db0-bf11-ecf53146ed0d" (UID: "46eff990-edaf-4db0-bf11-ecf53146ed0d"). InnerVolumeSpecName "kube-api-access-stpjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:27.648975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.648875    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-xks-ca" pod "46eff990-edaf-4db0-bf11-ecf53146ed0d" (UID: "46eff990-edaf-4db0-bf11-ecf53146ed0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:27.748763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.748730    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stpjd\" (UniqueName: \"kubernetes.io/projected/46eff990-edaf-4db0-bf11-ecf53146ed0d-kube-api-access-stpjd\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:27.748763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:27.748759    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46eff990-edaf-4db0-bf11-ecf53146ed0d-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:28.281701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.281667    4576 patch_prober.go:29] interesting pod/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:22:28.281895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.281728    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:22:28.408212 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.408134    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl" event={"ID":"46eff990-edaf-4db0-bf11-ecf53146ed0d","Type":"ContainerDied","Data":"bdf8fe105bf040a3ce7655a4e3deeb3ebaba39cdd3df09e7739bcdbaed5c35c9"}
Oct 30 01:22:28.408212 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.408173    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"
Oct 30 01:22:28.408212 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.408182    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf8fe105bf040a3ce7655a4e3deeb3ebaba39cdd3df09e7739bcdbaed5c35c9"
Oct 30 01:22:28.569413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.569357    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:28.615739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.615697    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"]
Oct 30 01:22:28.616653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.616634    4576 cpu_manager.go:414] "containerMap: removing container" podUID="46eff990-edaf-4db0-bf11-ecf53146ed0d" containerName="revoke-keys"
Oct 30 01:22:28.616750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.616663    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46eff990-edaf-4db0-bf11-ecf53146ed0d" containerName="revoke-keys"
Oct 30 01:22:28.616792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.616773    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="46eff990-edaf-4db0-bf11-ecf53146ed0d" containerName="revoke-keys"
Oct 30 01:22:28.620657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.620634    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.629254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.629211    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"]
Oct 30 01:22:28.638297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.638273    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:22:28.639135 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.639110    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:28.663329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.663094    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.663329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.663176    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.667405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.667359    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7q4d\" (UniqueName: \"kubernetes.io/projected/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-kube-api-access-f7q4d\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.768882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.768847    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.769083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.768901    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.769083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.768978    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7q4d\" (UniqueName: \"kubernetes.io/projected/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-kube-api-access-f7q4d\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.769669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.769647    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.771576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.771551    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.790223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.790197    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7q4d\" (UniqueName: \"kubernetes.io/projected/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-kube-api-access-f7q4d\") pod \"maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:28.946730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:28.946700    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:29.259747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.259715    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"]
Oct 30 01:22:29.305115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.305088    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:22:29.379711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.379678    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-kube-api-access-bklhg\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-kube-api-access-bklhg\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.379909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.379737    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-envoy\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.379909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.379760    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-token\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-token\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.379909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.379807    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-podinfo\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.379909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.379871    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istiod-ca-cert\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.380144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.380061    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-data\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.380144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.380099    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-socket\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.380144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.380135    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-certs\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.380380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.380174    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-credential-socket\") pod \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\" (UID: \"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c\") "
Oct 30 01:22:29.380604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.380580    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-data" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:29.381153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.380801    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-credential-socket" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:29.381153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.380983    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-socket" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:29.381153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.381003    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-certs" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:29.381153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.381102    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.381153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.381122    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.381533 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.381449    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istiod-ca-cert" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:29.422056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.422019    4576 generic.go:356] "Generic (PLEG): container finished" podID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerID="70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9" exitCode=0
Oct 30 01:22:29.422338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.422083    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"
Oct 30 01:22:29.422338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.422121    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" event={"ID":"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c","Type":"ContainerDied","Data":"70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9"}
Oct 30 01:22:29.422338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.422184    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt" event={"ID":"56f1033e-bfd7-49e9-9dcd-bb82d32dd11c","Type":"ContainerDied","Data":"e32b0c93d20a8299d726b87be81ab7f77368d47cedb92b0b5eb9ef193fdef045"}
Oct 30 01:22:29.422338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.422218    4576 scope.go:122] "RemoveContainer" containerID="70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9"
Oct 30 01:22:29.426119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.426092    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb" event={"ID":"7cf4c8b5-04b6-4346-b2d8-20f144b6853f","Type":"ContainerStarted","Data":"847e102dcabe33cb958e9629ed1c97df290be0ef6fcb06af48bba5d8aaa7d607"}
Oct 30 01:22:29.439834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.439302    4576 scope.go:122] "RemoveContainer" containerID="70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9"
Oct 30 01:22:29.439834 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:22:29.439707    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9\": container with ID starting with 70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9 not found: ID does not exist" containerID="70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9"
Oct 30 01:22:29.439834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.439745    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9"} err="failed to get container status \"70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9\": rpc error: code = NotFound desc = could not find container \"70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9\": container with ID starting with 70624d429e5539b74b66b1f4a740d3fe7978a61a7c0c3299cddfa0df197befb9 not found: ID does not exist"
Oct 30 01:22:29.468430 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.468319    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-token" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:29.468430 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.468312    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-envoy" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:29.468808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.468760    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-podinfo" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:29.468896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.468772    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-kube-api-access-bklhg" pod "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" (UID: "56f1033e-bfd7-49e9-9dcd-bb82d32dd11c"). InnerVolumeSpecName "kube-api-access-bklhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:29.484985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.484444    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklhg\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-kube-api-access-bklhg\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.484985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.484479    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.484985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.484494    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.484985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.484904    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.484985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.484918    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.484985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.484930    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.484985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.484942    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:29.496900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.496870    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"]
Oct 30 01:22:29.594597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.594083    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:29.779703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.779671    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"]
Oct 30 01:22:29.783832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:29.783793    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-43d152-openshift-default-585956f79f-5xngt"]
Oct 30 01:22:30.031116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:30.031036    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"]
Oct 30 01:22:30.031319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:30.031292    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy" containerID="cri-o://14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125" gracePeriod=30
Oct 30 01:22:30.110735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:30.110689    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" path="/var/lib/kubelet/pods/56f1033e-bfd7-49e9-9dcd-bb82d32dd11c/volumes"
Oct 30 01:22:30.356877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:30.356844    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:30.439014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:30.438958    4576 generic.go:356] "Generic (PLEG): container finished" podID="7cf4c8b5-04b6-4346-b2d8-20f144b6853f" containerID="9374b1fc07ef1625081ad0bfab170d38ee5d9145ec667ba2ce9d4802152931f0" exitCode=0
Oct 30 01:22:30.439014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:30.439002    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb" event={"ID":"7cf4c8b5-04b6-4346-b2d8-20f144b6853f","Type":"ContainerDied","Data":"9374b1fc07ef1625081ad0bfab170d38ee5d9145ec667ba2ce9d4802152931f0"}
Oct 30 01:22:31.629675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.629649    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:31.721295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.721197    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-xks-ca\") pod \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") "
Oct 30 01:22:31.721295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.721283    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-openshift-ca\") pod \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") "
Oct 30 01:22:31.721556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.721432    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-kube-api-access-f7q4d\" (UniqueName: \"kubernetes.io/projected/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-kube-api-access-f7q4d\") pod \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\" (UID: \"7cf4c8b5-04b6-4346-b2d8-20f144b6853f\") "
Oct 30 01:22:31.721784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.721755    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-openshift-ca" pod "7cf4c8b5-04b6-4346-b2d8-20f144b6853f" (UID: "7cf4c8b5-04b6-4346-b2d8-20f144b6853f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:31.722538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.722511    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:31.723726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.723702    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-xks-ca" pod "7cf4c8b5-04b6-4346-b2d8-20f144b6853f" (UID: "7cf4c8b5-04b6-4346-b2d8-20f144b6853f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:31.723891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.723876    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-kube-api-access-f7q4d" pod "7cf4c8b5-04b6-4346-b2d8-20f144b6853f" (UID: "7cf4c8b5-04b6-4346-b2d8-20f144b6853f"). InnerVolumeSpecName "kube-api-access-f7q4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:31.824482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.824444    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:31.824482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:31.824476    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7q4d\" (UniqueName: \"kubernetes.io/projected/7cf4c8b5-04b6-4346-b2d8-20f144b6853f-kube-api-access-f7q4d\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:32.456681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:32.456654    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"
Oct 30 01:22:32.456861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:32.456693    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb" event={"ID":"7cf4c8b5-04b6-4346-b2d8-20f144b6853f","Type":"ContainerDied","Data":"847e102dcabe33cb958e9629ed1c97df290be0ef6fcb06af48bba5d8aaa7d607"}
Oct 30 01:22:32.456861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:32.456749    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="847e102dcabe33cb958e9629ed1c97df290be0ef6fcb06af48bba5d8aaa7d607"
Oct 30 01:22:32.612641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:32.612610    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:34.197905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:34.197873    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:35.262907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.262871    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:35.373061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.373020    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:35.379046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.379019    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:22:35.484945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.484908    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-podinfo\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.485148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.485065    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba194d2d-7baf-4923-8084-e551b8e2125f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba194d2d-7baf-4923-8084-e551b8e2125f-istiod-ca-cert\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.485148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.485124    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-credential-socket\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.485365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.485287    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-kube-api-access-wzltp\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-kube-api-access-wzltp\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.485365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.485337    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-socket\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.485501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.485410    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-envoy\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.485501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.485457    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-data\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.485609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.485519    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-token\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-token\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.485609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.485549    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-certs\") pod \"ba194d2d-7baf-4923-8084-e551b8e2125f\" (UID: \"ba194d2d-7baf-4923-8084-e551b8e2125f\") "
Oct 30 01:22:35.486074 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.486045    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-socket" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:35.486952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.486364    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-data" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:35.486952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.486901    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-credential-socket" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:35.487268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.487243    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-certs" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:35.488424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.487433    4576 generic.go:356] "Generic (PLEG): container finished" podID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerID="14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125" exitCode=0
Oct 30 01:22:35.488424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.487539    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" event={"ID":"ba194d2d-7baf-4923-8084-e551b8e2125f","Type":"ContainerDied","Data":"14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125"}
Oct 30 01:22:35.488424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.487587    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc" event={"ID":"ba194d2d-7baf-4923-8084-e551b8e2125f","Type":"ContainerDied","Data":"8692e37bacb9d8885992593ce5ab332b95e96bcec8ba402eeeadcee97ad3719b"}
Oct 30 01:22:35.488424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.487598    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"
Oct 30 01:22:35.488424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.487614    4576 scope.go:122] "RemoveContainer" containerID="14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125"
Oct 30 01:22:35.489075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489047    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba194d2d-7baf-4923-8084-e551b8e2125f-istiod-ca-cert" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:35.489279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489203    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-podinfo" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:35.489279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489222    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-kube-api-access-wzltp" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "kube-api-access-wzltp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:35.489712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489692    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzltp\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-kube-api-access-wzltp\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.489793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489715    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.489793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489730    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.489793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489742    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.489793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489754    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.489793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489766    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba194d2d-7baf-4923-8084-e551b8e2125f-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.489793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.489778    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.496016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.495758    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-token" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:35.502519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.502388    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-envoy" pod "ba194d2d-7baf-4923-8084-e551b8e2125f" (UID: "ba194d2d-7baf-4923-8084-e551b8e2125f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:35.530141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.530116    4576 scope.go:122] "RemoveContainer" containerID="14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125"
Oct 30 01:22:35.530483 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:22:35.530460    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125\": container with ID starting with 14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125 not found: ID does not exist" containerID="14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125"
Oct 30 01:22:35.530556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.530493    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125"} err="failed to get container status \"14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125\": rpc error: code = NotFound desc = could not find container \"14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125\": container with ID starting with 14f6925e63f6fba5251af9b18651b5e013596df203ce0d34c17393b9ac027125 not found: ID does not exist"
Oct 30 01:22:35.591487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.591412    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.591487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.591437    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba194d2d-7baf-4923-8084-e551b8e2125f-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:35.839685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.839652    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"]
Oct 30 01:22:35.853127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.853048    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ca324d-openshift-default-788bf89f9f-fr2gc"]
Oct 30 01:22:35.876567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:35.876524    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:36.116669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:36.116567    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" path="/var/lib/kubelet/pods/ba194d2d-7baf-4923-8084-e551b8e2125f/volumes"
Oct 30 01:22:37.567621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:37.567587    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:39.059211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:39.059171    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:40.269405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:40.269367    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:40.648851 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:40.648758    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:41.844616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:41.844585    4576 scope.go:122] "RemoveContainer" containerID="f07add62a8170b2368717fe06d59873ce056235023b101512a293540a80aeb9b"
Oct 30 01:22:41.855415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:41.855389    4576 scope.go:122] "RemoveContainer" containerID="9fadee815c72a22d9c947fe3cd8e9154105291fa3765783798cf813278b9df2d"
Oct 30 01:22:41.865899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:41.865878    4576 scope.go:122] "RemoveContainer" containerID="5710fa627c8543ddcacf684bae9d6b11bae9610056adc28fc26f57db0ad65d28"
Oct 30 01:22:41.877733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:41.877704    4576 scope.go:122] "RemoveContainer" containerID="f3d0ea39a965c618b2c8f49ee1118f628e63c59f4facc7e270e15d693df99f04"
Oct 30 01:22:41.888075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:41.888052    4576 scope.go:122] "RemoveContainer" containerID="9ff591e6424e5d06167c511a5d214fd13052a2d2c51c7bdbefb1ca206cf51cdb"
Oct 30 01:22:41.897837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:41.897815    4576 scope.go:122] "RemoveContainer" containerID="9359f9572e1380283b66b1267102a51282de27ab948f29d4b8045a0cb64d5167"
Oct 30 01:22:41.907490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:41.907465    4576 scope.go:122] "RemoveContainer" containerID="96e55be0aa0b13f237a5c017b0adaa795eded2af8d3dfd06f13c5858d5f88a30"
Oct 30 01:22:41.917764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:41.917744    4576 scope.go:122] "RemoveContainer" containerID="f0ea12f71beaaf037a3f828e6a9fa94a9e05035eb3cdae8afe115e54b7cb9072"
Oct 30 01:22:42.216674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:42.216635    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:43.833008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:43.832968    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:22:43.939982 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:43.939945    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:44.521691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.521632    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main" containerID="cri-o://bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720" gracePeriod=45
Oct 30 01:22:44.677808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.677772    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:44.804314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.804286    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 01:22:44.860157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.860120    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-dshm\") pod \"b8ea057e-546a-43b4-b257-74dd27f07b08\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") "
Oct 30 01:22:44.860704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.860186    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-tmp-dir\") pod \"b8ea057e-546a-43b4-b257-74dd27f07b08\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") "
Oct 30 01:22:44.860704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.860249    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-home\") pod \"b8ea057e-546a-43b4-b257-74dd27f07b08\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") "
Oct 30 01:22:44.860704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.860399    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ea057e-546a-43b4-b257-74dd27f07b08-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea057e-546a-43b4-b257-74dd27f07b08-tls-certs\") pod \"b8ea057e-546a-43b4-b257-74dd27f07b08\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") "
Oct 30 01:22:44.860704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.860597    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ea057e-546a-43b4-b257-74dd27f07b08-kube-api-access-mv7dw\" (UniqueName: \"kubernetes.io/projected/b8ea057e-546a-43b4-b257-74dd27f07b08-kube-api-access-mv7dw\") pod \"b8ea057e-546a-43b4-b257-74dd27f07b08\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") "
Oct 30 01:22:44.860704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.860670    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-model-cache\") pod \"b8ea057e-546a-43b4-b257-74dd27f07b08\" (UID: \"b8ea057e-546a-43b4-b257-74dd27f07b08\") "
Oct 30 01:22:44.861769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.861736    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-model-cache" pod "b8ea057e-546a-43b4-b257-74dd27f07b08" (UID: "b8ea057e-546a-43b4-b257-74dd27f07b08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:44.862381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.862342    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-home" pod "b8ea057e-546a-43b4-b257-74dd27f07b08" (UID: "b8ea057e-546a-43b4-b257-74dd27f07b08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:44.863441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.863406    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-tmp-dir" pod "b8ea057e-546a-43b4-b257-74dd27f07b08" (UID: "b8ea057e-546a-43b4-b257-74dd27f07b08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:44.864906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.864878    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-dshm" pod "b8ea057e-546a-43b4-b257-74dd27f07b08" (UID: "b8ea057e-546a-43b4-b257-74dd27f07b08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:44.865619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.865598    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ea057e-546a-43b4-b257-74dd27f07b08-kube-api-access-mv7dw" pod "b8ea057e-546a-43b4-b257-74dd27f07b08" (UID: "b8ea057e-546a-43b4-b257-74dd27f07b08"). InnerVolumeSpecName "kube-api-access-mv7dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:44.865890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.865864    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ea057e-546a-43b4-b257-74dd27f07b08-tls-certs" pod "b8ea057e-546a-43b4-b257-74dd27f07b08" (UID: "b8ea057e-546a-43b4-b257-74dd27f07b08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:44.898534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.898493    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:44.963544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.963415    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7dw\" (UniqueName: \"kubernetes.io/projected/b8ea057e-546a-43b4-b257-74dd27f07b08-kube-api-access-mv7dw\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:44.963544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.963450    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:44.963544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.963465    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:44.963544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.963476    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:44.963544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.963487    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ea057e-546a-43b4-b257-74dd27f07b08-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:44.963544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:44.963501    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ea057e-546a-43b4-b257-74dd27f07b08-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:45.576713 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.576627    4576 generic.go:356] "Generic (PLEG): container finished" podID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerID="bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720" exitCode=0
Oct 30 01:22:45.577462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.577384    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" event={"ID":"b8ea057e-546a-43b4-b257-74dd27f07b08","Type":"ContainerDied","Data":"bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720"}
Oct 30 01:22:45.577682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.577661    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28" event={"ID":"b8ea057e-546a-43b4-b257-74dd27f07b08","Type":"ContainerDied","Data":"b32aac6623b669362d1fc2e9625d8329a056026088b110911d8505ddf846baf0"}
Oct 30 01:22:45.577800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.577788    4576 scope.go:122] "RemoveContainer" containerID="bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720"
Oct 30 01:22:45.579506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.578050    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"
Oct 30 01:22:45.610668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.610633    4576 scope.go:122] "RemoveContainer" containerID="bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720"
Oct 30 01:22:45.611152 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:22:45.611073    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720\": container with ID starting with bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720 not found: ID does not exist" containerID="bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720"
Oct 30 01:22:45.611152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.611113    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720"} err="failed to get container status \"bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720\": rpc error: code = NotFound desc = could not find container \"bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720\": container with ID starting with bff3ff5fd1c347bac7e02ce3c6217ee9f672b764451e2cfc27a8b6e9f25b8720 not found: ID does not exist"
Oct 30 01:22:45.647681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.647644    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"]
Oct 30 01:22:45.661497 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:45.661466    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3c2596/simulator-w0-3c2596-kserve-9569fc9f5-dzf28"]
Oct 30 01:22:46.123538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:46.123502    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" path="/var/lib/kubelet/pods/b8ea057e-546a-43b4-b257-74dd27f07b08/volumes"
Oct 30 01:22:46.849406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:46.849363    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:48.111272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111202    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"]
Oct 30 01:22:48.111893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111877    4576 cpu_manager.go:414] "containerMap: removing container" podUID="7cf4c8b5-04b6-4346-b2d8-20f144b6853f" containerName="revoke-keys"
Oct 30 01:22:48.111949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111895    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf4c8b5-04b6-4346-b2d8-20f144b6853f" containerName="revoke-keys"
Oct 30 01:22:48.111949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111910    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy"
Oct 30 01:22:48.111949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111915    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy"
Oct 30 01:22:48.111949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111933    4576 cpu_manager.go:414] "containerMap: removing container" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy"
Oct 30 01:22:48.111949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111938    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy"
Oct 30 01:22:48.112143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111958    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main"
Oct 30 01:22:48.112143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.111963    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main"
Oct 30 01:22:48.112143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.112094    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf4c8b5-04b6-4346-b2d8-20f144b6853f" containerName="revoke-keys"
Oct 30 01:22:48.112143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.112108    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba194d2d-7baf-4923-8084-e551b8e2125f" containerName="istio-proxy"
Oct 30 01:22:48.112143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.112121    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f1033e-bfd7-49e9-9dcd-bb82d32dd11c" containerName="istio-proxy"
Oct 30 01:22:48.112143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.112135    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ea057e-546a-43b4-b257-74dd27f07b08" containerName="main"
Oct 30 01:22:48.117555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.117533    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.123814 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.123789    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"]
Oct 30 01:22:48.133951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.133918    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:48.147848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.147821    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:22:48.185579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.185546    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.185981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.185956    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bbq\" (UniqueName: \"kubernetes.io/projected/057947eb-9d61-4733-852f-2b22e8bc9666-kube-api-access-z7bbq\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.186095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.186038    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.287421 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.287388    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bbq\" (UniqueName: \"kubernetes.io/projected/057947eb-9d61-4733-852f-2b22e8bc9666-kube-api-access-z7bbq\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.287606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.287461    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.287606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.287578    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.288445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.288415    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.290083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.290058    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.308382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.308351    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bbq\" (UniqueName: \"kubernetes.io/projected/057947eb-9d61-4733-852f-2b22e8bc9666-kube-api-access-z7bbq\") pod \"maas-api-revoke-sub-1f989bb2-djm28\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.384180 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:22:48.384089    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:22:48.435205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.435168    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:48.745889 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:22:48.745858    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057947eb_9d61_4733_852f_2b22e8bc9666.slice/crio-c645f99eb3bf8e40e1c1278f1fc5f823bb059d709932b718a6187a44acebc437 WatchSource:0}: Error finding container c645f99eb3bf8e40e1c1278f1fc5f823bb059d709932b718a6187a44acebc437: Status 404 returned error can't find the container with id c645f99eb3bf8e40e1c1278f1fc5f823bb059d709932b718a6187a44acebc437
Oct 30 01:22:48.746056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:48.746034    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"]
Oct 30 01:22:49.234322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:49.234288    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:49.615706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:49.615669    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28" event={"ID":"057947eb-9d61-4733-852f-2b22e8bc9666","Type":"ContainerStarted","Data":"069df0c83e1d2d0ccb4bcf0021efde98717e6557171cc1c42770354054cabba6"}
Oct 30 01:22:49.615706 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:49.615733    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28" event={"ID":"057947eb-9d61-4733-852f-2b22e8bc9666","Type":"ContainerStarted","Data":"c645f99eb3bf8e40e1c1278f1fc5f823bb059d709932b718a6187a44acebc437"}
Oct 30 01:22:49.658815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:49.658751    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28" podStartSLOduration=1.6587338969999998 podStartE2EDuration="1.658733897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:49.65631035 +0000 UTC m=+2649.727634196" watchObservedRunningTime="2026-10-30 01:22:49.658733897 +0000 UTC m=+2649.730057732"
Oct 30 01:22:50.634832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:50.634792    4576 generic.go:356] "Generic (PLEG): container finished" podID="057947eb-9d61-4733-852f-2b22e8bc9666" containerID="069df0c83e1d2d0ccb4bcf0021efde98717e6557171cc1c42770354054cabba6" exitCode=0
Oct 30 01:22:50.635320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:50.634891    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28" event={"ID":"057947eb-9d61-4733-852f-2b22e8bc9666","Type":"ContainerDied","Data":"069df0c83e1d2d0ccb4bcf0021efde98717e6557171cc1c42770354054cabba6"}
Oct 30 01:22:51.285821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.285780    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:51.816274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.816246    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:51.947547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.947517    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-openshift-ca\") pod \"057947eb-9d61-4733-852f-2b22e8bc9666\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") "
Oct 30 01:22:51.947779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.947766    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-xks-ca\") pod \"057947eb-9d61-4733-852f-2b22e8bc9666\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") "
Oct 30 01:22:51.947829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.947792    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057947eb-9d61-4733-852f-2b22e8bc9666-kube-api-access-z7bbq\" (UniqueName: \"kubernetes.io/projected/057947eb-9d61-4733-852f-2b22e8bc9666-kube-api-access-z7bbq\") pod \"057947eb-9d61-4733-852f-2b22e8bc9666\" (UID: \"057947eb-9d61-4733-852f-2b22e8bc9666\") "
Oct 30 01:22:51.947900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.947879    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-openshift-ca" pod "057947eb-9d61-4733-852f-2b22e8bc9666" (UID: "057947eb-9d61-4733-852f-2b22e8bc9666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:51.948348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.948322    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:51.950126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.950091    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-xks-ca" pod "057947eb-9d61-4733-852f-2b22e8bc9666" (UID: "057947eb-9d61-4733-852f-2b22e8bc9666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:51.950126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:51.950113    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057947eb-9d61-4733-852f-2b22e8bc9666-kube-api-access-z7bbq" pod "057947eb-9d61-4733-852f-2b22e8bc9666" (UID: "057947eb-9d61-4733-852f-2b22e8bc9666"). InnerVolumeSpecName "kube-api-access-z7bbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:52.049987 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.049947    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057947eb-9d61-4733-852f-2b22e8bc9666-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:52.049987 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.049984    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7bbq\" (UniqueName: \"kubernetes.io/projected/057947eb-9d61-4733-852f-2b22e8bc9666-kube-api-access-z7bbq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:52.232329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.232194    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:52.530627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.530547    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"]
Oct 30 01:22:52.565436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.565392    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"]
Oct 30 01:22:52.659088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.659058    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"
Oct 30 01:22:52.659088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.659071    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28" event={"ID":"057947eb-9d61-4733-852f-2b22e8bc9666","Type":"ContainerDied","Data":"c645f99eb3bf8e40e1c1278f1fc5f823bb059d709932b718a6187a44acebc437"}
Oct 30 01:22:52.659356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.659122    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c645f99eb3bf8e40e1c1278f1fc5f823bb059d709932b718a6187a44acebc437"
Oct 30 01:22:52.702778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.702746    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:52.816829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:52.816742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:53.079880 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.079773    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:53.347420 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.347319    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:53.488102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.488069    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"]
Oct 30 01:22:53.488977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.488954    4576 cpu_manager.go:414] "containerMap: removing container" podUID="057947eb-9d61-4733-852f-2b22e8bc9666" containerName="revoke-keys"
Oct 30 01:22:53.488977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.488975    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057947eb-9d61-4733-852f-2b22e8bc9666" containerName="revoke-keys"
Oct 30 01:22:53.489192 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.489101    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="057947eb-9d61-4733-852f-2b22e8bc9666" containerName="revoke-keys"
Oct 30 01:22:53.495252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.494519    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.519258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.513753    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"]
Oct 30 01:22:53.521361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.521101    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-11e7a9-openshift-default-dockercfg-pccwv\""
Oct 30 01:22:53.568001 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.567796    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:53.571758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.571730    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-token\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.571889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.571792    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-podinfo\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.571974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.571937    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-data\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.572066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.572049    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-envoy\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.572121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.572096    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-socket\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.572310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.572290    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-credential-socket\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.572389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.572331    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcxh\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-kube-api-access-qxcxh\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.572545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.572523    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-certs\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.572613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.572558    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bca5e0-3000-47b1-a675-e717c79a40a7-istiod-ca-cert\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677384    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-envoy\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677463    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-socket\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677589    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-credential-socket\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677620    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcxh\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-kube-api-access-qxcxh\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677770    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-certs\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677794    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bca5e0-3000-47b1-a675-e717c79a40a7-istiod-ca-cert\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677892    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-token\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677932    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-podinfo\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.677986    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-data\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.678932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.678497    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-data\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.689255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.684900    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-certs\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.689255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.685156    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-socket\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.689255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.685433    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-credential-socket\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.689255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.685853    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bca5e0-3000-47b1-a675-e717c79a40a7-istiod-ca-cert\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.780296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.752438    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-token\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.780296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.769283    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcxh\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-kube-api-access-qxcxh\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.780296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.769848    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-envoy\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.800460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.784551    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-podinfo\") pod \"e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") " pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:53.839617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:53.838354    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:54.266496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.266462    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"]
Oct 30 01:22:54.266920 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:22:54.266892    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bca5e0_3000_47b1_a675_e717c79a40a7.slice/crio-aa761a7059d2a84e4eb3ad3ba120af7e93b681d42a9f580f349e82d6d57fa5e7 WatchSource:0}: Error finding container aa761a7059d2a84e4eb3ad3ba120af7e93b681d42a9f580f349e82d6d57fa5e7: Status 404 returned error can't find the container with id aa761a7059d2a84e4eb3ad3ba120af7e93b681d42a9f580f349e82d6d57fa5e7
Oct 30 01:22:54.270472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.270429    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:22:54.270573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.270526    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:22:54.270615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.270568    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:22:54.280400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.280369    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:54.733834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.733793    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" event={"ID":"66bca5e0-3000-47b1-a675-e717c79a40a7","Type":"ContainerStarted","Data":"f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29"}
Oct 30 01:22:54.734049 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.733866    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" event={"ID":"66bca5e0-3000-47b1-a675-e717c79a40a7","Type":"ContainerStarted","Data":"aa761a7059d2a84e4eb3ad3ba120af7e93b681d42a9f580f349e82d6d57fa5e7"}
Oct 30 01:22:54.781193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.781137    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podStartSLOduration=1.781119124 podStartE2EDuration="1.781119124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:54.77817247 +0000 UTC m=+2654.849496300" watchObservedRunningTime="2026-10-30 01:22:54.781119124 +0000 UTC m=+2654.852442959"
Oct 30 01:22:54.841078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:54.841044    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:55.331155 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:55.331110    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:55.847622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:55.847596    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:56.750249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:56.750209    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:56.751604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:56.751575    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:22:57.065053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.064933    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:57.343309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.343046    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:57.531056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.531014    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6" podUID="2570a7e8-ff3b-4b05-a333-08cec97a0363" containerName="payload-processing" containerID="cri-o://6854cbfd2573a62c29784ab219dbcd28bbb9fb021cbbc358c560bee3b1cf03c8" gracePeriod=25
Oct 30 01:22:57.566714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.566680    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8" podUID="12f77b7f-4d8b-47ea-bc75-5b3d70a63b05" containerName="payload-pre-processing" containerID="cri-o://e54e05199ecab15f1bba9048fcbf822d484da479b3a492bb0d254aab0e2fdc72" gracePeriod=25
Oct 30 01:22:57.768256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.765882    4576 generic.go:356] "Generic (PLEG): container finished" podID="2570a7e8-ff3b-4b05-a333-08cec97a0363" containerID="6854cbfd2573a62c29784ab219dbcd28bbb9fb021cbbc358c560bee3b1cf03c8" exitCode=0
Oct 30 01:22:57.768256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.765975    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6" event={"ID":"2570a7e8-ff3b-4b05-a333-08cec97a0363","Type":"ContainerDied","Data":"6854cbfd2573a62c29784ab219dbcd28bbb9fb021cbbc358c560bee3b1cf03c8"}
Oct 30 01:22:57.768256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.767981    4576 generic.go:356] "Generic (PLEG): container finished" podID="12f77b7f-4d8b-47ea-bc75-5b3d70a63b05" containerID="e54e05199ecab15f1bba9048fcbf822d484da479b3a492bb0d254aab0e2fdc72" exitCode=0
Oct 30 01:22:57.782594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.780093    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8" event={"ID":"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05","Type":"ContainerDied","Data":"e54e05199ecab15f1bba9048fcbf822d484da479b3a492bb0d254aab0e2fdc72"}
Oct 30 01:22:57.856347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.856298    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:22:57.925030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.924874    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:22:57.994962 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.994932    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2570a7e8-ff3b-4b05-a333-08cec97a0363-kube-api-access-42ctt\" (UniqueName: \"kubernetes.io/projected/2570a7e8-ff3b-4b05-a333-08cec97a0363-kube-api-access-42ctt\") pod \"2570a7e8-ff3b-4b05-a333-08cec97a0363\" (UID: \"2570a7e8-ff3b-4b05-a333-08cec97a0363\") "
Oct 30 01:22:57.995172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.995157    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2570a7e8-ff3b-4b05-a333-08cec97a0363-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2570a7e8-ff3b-4b05-a333-08cec97a0363-plugins-config-volume\") pod \"2570a7e8-ff3b-4b05-a333-08cec97a0363\" (UID: \"2570a7e8-ff3b-4b05-a333-08cec97a0363\") "
Oct 30 01:22:57.995561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.995528    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2570a7e8-ff3b-4b05-a333-08cec97a0363-plugins-config-volume" pod "2570a7e8-ff3b-4b05-a333-08cec97a0363" (UID: "2570a7e8-ff3b-4b05-a333-08cec97a0363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:57.995942 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.995923    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2570a7e8-ff3b-4b05-a333-08cec97a0363-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:57.997445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:57.997417    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2570a7e8-ff3b-4b05-a333-08cec97a0363-kube-api-access-42ctt" pod "2570a7e8-ff3b-4b05-a333-08cec97a0363" (UID: "2570a7e8-ff3b-4b05-a333-08cec97a0363"). InnerVolumeSpecName "kube-api-access-42ctt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:58.096631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.096594    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-plugins-config-volume\") pod \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\" (UID: \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\") "
Oct 30 01:22:58.097091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.096817    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-kube-api-access-w5ksf\" (UniqueName: \"kubernetes.io/projected/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-kube-api-access-w5ksf\") pod \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\" (UID: \"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05\") "
Oct 30 01:22:58.097091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.096955    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-plugins-config-volume" pod "12f77b7f-4d8b-47ea-bc75-5b3d70a63b05" (UID: "12f77b7f-4d8b-47ea-bc75-5b3d70a63b05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:58.097516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.097495    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:58.097630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.097518    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ctt\" (UniqueName: \"kubernetes.io/projected/2570a7e8-ff3b-4b05-a333-08cec97a0363-kube-api-access-42ctt\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:58.099193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.099173    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-kube-api-access-w5ksf" pod "12f77b7f-4d8b-47ea-bc75-5b3d70a63b05" (UID: "12f77b7f-4d8b-47ea-bc75-5b3d70a63b05"). InnerVolumeSpecName "kube-api-access-w5ksf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:58.199675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.199634    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5ksf\" (UniqueName: \"kubernetes.io/projected/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05-kube-api-access-w5ksf\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:22:58.450128 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.450045    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"]
Oct 30 01:22:58.450846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.450828    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2570a7e8-ff3b-4b05-a333-08cec97a0363" containerName="payload-processing"
Oct 30 01:22:58.450893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.450849    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2570a7e8-ff3b-4b05-a333-08cec97a0363" containerName="payload-processing"
Oct 30 01:22:58.450893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.450891    4576 cpu_manager.go:414] "containerMap: removing container" podUID="12f77b7f-4d8b-47ea-bc75-5b3d70a63b05" containerName="payload-pre-processing"
Oct 30 01:22:58.450963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.450896    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f77b7f-4d8b-47ea-bc75-5b3d70a63b05" containerName="payload-pre-processing"
Oct 30 01:22:58.451002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.450995    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2570a7e8-ff3b-4b05-a333-08cec97a0363" containerName="payload-processing"
Oct 30 01:22:58.451047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.451010    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f77b7f-4d8b-47ea-bc75-5b3d70a63b05" containerName="payload-pre-processing"
Oct 30 01:22:58.455095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.455063    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.472787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.472760    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"]
Oct 30 01:22:58.483254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.482312    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:22:58.483254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.482378    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:22:58.605132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.605098    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.605349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.605267    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.605486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.605468    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvgs\" (UniqueName: \"kubernetes.io/projected/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-kube-api-access-nlvgs\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.706810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.706717    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvgs\" (UniqueName: \"kubernetes.io/projected/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-kube-api-access-nlvgs\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.706991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.706921    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.707065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.707000    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.708085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.708055    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.724315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.709973    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.727793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.727769    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvgs\" (UniqueName: \"kubernetes.io/projected/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-kube-api-access-nlvgs\") pod \"maas-api-revoke-sub-1725b6af-6l6rs\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.742092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.742058    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:58.770767 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.770735    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:22:58.778943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.778912    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6" event={"ID":"2570a7e8-ff3b-4b05-a333-08cec97a0363","Type":"ContainerDied","Data":"f3d27a6eaf9564c8198df2db6006d1d708068553b366c2e3838d841c013727a0"}
Oct 30 01:22:58.779087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.778981    4576 scope.go:122] "RemoveContainer" containerID="6854cbfd2573a62c29784ab219dbcd28bbb9fb021cbbc358c560bee3b1cf03c8"
Oct 30 01:22:58.779176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.779157    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"
Oct 30 01:22:58.781308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.780752    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"
Oct 30 01:22:58.781308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.780794    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8" event={"ID":"12f77b7f-4d8b-47ea-bc75-5b3d70a63b05","Type":"ContainerDied","Data":"22b6b3c2be6910c420fb297bf5dadb77c69749b7a4816b7a3494ec9b9f01f3fd"}
Oct 30 01:22:58.817271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.817213    4576 scope.go:122] "RemoveContainer" containerID="e54e05199ecab15f1bba9048fcbf822d484da479b3a492bb0d254aab0e2fdc72"
Oct 30 01:22:58.857020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.856952    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:58.860332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.860274    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"]
Oct 30 01:22:58.866336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.866309    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-11fbe8-5d89cff8c8-fsjn6"]
Oct 30 01:22:58.908491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.908462    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"]
Oct 30 01:22:58.914866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.914832    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"]
Oct 30 01:22:58.929735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.929705    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fbe8-7bfb8f97c5-r74p8"]
Oct 30 01:22:58.929967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.929826    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:22:58.938594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.937422    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"]
Oct 30 01:22:58.949311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:58.949287    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-11e7a9\""
Oct 30 01:22:59.014385 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.007597    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-11e7a9-dockercfg-d5h6f\""
Oct 30 01:22:59.054840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.054810    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"]
Oct 30 01:22:59.065287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.065211    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"]
Oct 30 01:22:59.065626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.065607    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:22:59.174549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.163333    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt56t\" (UniqueName: \"kubernetes.io/projected/00d9aef7-2d8f-4e94-b364-82d203e38739-kube-api-access-nt56t\") pod \"payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt\" (UID: \"00d9aef7-2d8f-4e94-b364-82d203e38739\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:22:59.174549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.163456    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndss\" (UniqueName: \"kubernetes.io/projected/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-kube-api-access-2ndss\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn\" (UID: \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:22:59.174549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.163490    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d9aef7-2d8f-4e94-b364-82d203e38739-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt\" (UID: \"00d9aef7-2d8f-4e94-b364-82d203e38739\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:22:59.174549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.163630    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn\" (UID: \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:22:59.194017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.193939    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"]
Oct 30 01:22:59.201138 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:22:59.201111    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cd3ef8_840a_4de0_ae7c_407eeaf59a3a.slice/crio-c00380eccdbdb9411651a64bc571af512dd67c0d1ee3f088e28c37efc00f767a WatchSource:0}: Error finding container c00380eccdbdb9411651a64bc571af512dd67c0d1ee3f088e28c37efc00f767a: Status 404 returned error can't find the container with id c00380eccdbdb9411651a64bc571af512dd67c0d1ee3f088e28c37efc00f767a
Oct 30 01:22:59.278826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.278732    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt56t\" (UniqueName: \"kubernetes.io/projected/00d9aef7-2d8f-4e94-b364-82d203e38739-kube-api-access-nt56t\") pod \"payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt\" (UID: \"00d9aef7-2d8f-4e94-b364-82d203e38739\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:22:59.278994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.278883    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndss\" (UniqueName: \"kubernetes.io/projected/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-kube-api-access-2ndss\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn\" (UID: \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:22:59.278994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.278920    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d9aef7-2d8f-4e94-b364-82d203e38739-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt\" (UID: \"00d9aef7-2d8f-4e94-b364-82d203e38739\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:22:59.279259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.279208    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn\" (UID: \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:22:59.280152 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.280124    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn\" (UID: \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:22:59.281096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.281067    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d9aef7-2d8f-4e94-b364-82d203e38739-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt\" (UID: \"00d9aef7-2d8f-4e94-b364-82d203e38739\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:22:59.305002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.304960    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt56t\" (UniqueName: \"kubernetes.io/projected/00d9aef7-2d8f-4e94-b364-82d203e38739-kube-api-access-nt56t\") pod \"payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt\" (UID: \"00d9aef7-2d8f-4e94-b364-82d203e38739\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:22:59.305205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.304959    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndss\" (UniqueName: \"kubernetes.io/projected/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-kube-api-access-2ndss\") pod \"payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn\" (UID: \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:22:59.461637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.461601    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:22:59.491335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.491303    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:22:59.549043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.549001    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:22:59.792062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.791977    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs" event={"ID":"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a","Type":"ContainerStarted","Data":"78b1e96e90880fa732206dec2826a97848079a240acb12e4ffe8db412804708c"}
Oct 30 01:22:59.792276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.792060    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs" event={"ID":"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a","Type":"ContainerStarted","Data":"c00380eccdbdb9411651a64bc571af512dd67c0d1ee3f088e28c37efc00f767a"}
Oct 30 01:22:59.836207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.836150    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs" podStartSLOduration=1.8361349009999999 podStartE2EDuration="1.836134901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:59.833113555 +0000 UTC m=+2659.904437398" watchObservedRunningTime="2026-10-30 01:22:59.836134901 +0000 UTC m=+2659.907458735"
Oct 30 01:22:59.854690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.854657    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"]
Oct 30 01:22:59.855913 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:22:59.855890    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d9aef7_2d8f_4e94_b364_82d203e38739.slice/crio-cdd960fc557439dc9889b154fa864be37ec41ebd2b576f2f388b5c278864d9bc WatchSource:0}: Error finding container cdd960fc557439dc9889b154fa864be37ec41ebd2b576f2f388b5c278864d9bc: Status 404 returned error can't find the container with id cdd960fc557439dc9889b154fa864be37ec41ebd2b576f2f388b5c278864d9bc
Oct 30 01:22:59.911734 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:22:59.911708    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec04e918_bc46_4d4f_b2fa_735b886ea3b5.slice/crio-aecbe25bec883bcdec3200d0aeab1a613fc86e493b80551551e656797c8001db WatchSource:0}: Error finding container aecbe25bec883bcdec3200d0aeab1a613fc86e493b80551551e656797c8001db: Status 404 returned error can't find the container with id aecbe25bec883bcdec3200d0aeab1a613fc86e493b80551551e656797c8001db
Oct 30 01:22:59.912577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:22:59.912551    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"]
Oct 30 01:23:00.110171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.110074    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f77b7f-4d8b-47ea-bc75-5b3d70a63b05" path="/var/lib/kubelet/pods/12f77b7f-4d8b-47ea-bc75-5b3d70a63b05/volumes"
Oct 30 01:23:00.110675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.110651    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2570a7e8-ff3b-4b05-a333-08cec97a0363" path="/var/lib/kubelet/pods/2570a7e8-ff3b-4b05-a333-08cec97a0363/volumes"
Oct 30 01:23:00.804577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.804536    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt" event={"ID":"00d9aef7-2d8f-4e94-b364-82d203e38739","Type":"ContainerStarted","Data":"d3ad6c42b2165fe5ae3fdf5ab00eb7d6b8dc18dc630e07f703149a1f12001f13"}
Oct 30 01:23:00.805019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.804601    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt" event={"ID":"00d9aef7-2d8f-4e94-b364-82d203e38739","Type":"ContainerStarted","Data":"cdd960fc557439dc9889b154fa864be37ec41ebd2b576f2f388b5c278864d9bc"}
Oct 30 01:23:00.805019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.804630    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:23:00.806149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.806121    4576 generic.go:356] "Generic (PLEG): container finished" podID="94cd3ef8-840a-4de0-ae7c-407eeaf59a3a" containerID="78b1e96e90880fa732206dec2826a97848079a240acb12e4ffe8db412804708c" exitCode=0
Oct 30 01:23:00.806333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.806204    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs" event={"ID":"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a","Type":"ContainerDied","Data":"78b1e96e90880fa732206dec2826a97848079a240acb12e4ffe8db412804708c"}
Oct 30 01:23:00.808008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.807951    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn" event={"ID":"ec04e918-bc46-4d4f-b2fa-735b886ea3b5","Type":"ContainerStarted","Data":"bb97ca754f09fb217c3f31a2bdb60cf861d525d5e7bce2338cbc8a02003d3822"}
Oct 30 01:23:00.808008 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.807986    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn" event={"ID":"ec04e918-bc46-4d4f-b2fa-735b886ea3b5","Type":"ContainerStarted","Data":"aecbe25bec883bcdec3200d0aeab1a613fc86e493b80551551e656797c8001db"}
Oct 30 01:23:00.808160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.808071    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:23:00.846274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.846187    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt" podStartSLOduration=2.846167447 podStartE2EDuration="2.846167447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:00.845969008 +0000 UTC m=+2660.917292843" watchObservedRunningTime="2026-10-30 01:23:00.846167447 +0000 UTC m=+2660.917491283"
Oct 30 01:23:00.932957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:00.932896    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn" podStartSLOduration=2.93287599 podStartE2EDuration="2.93287599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:00.926079379 +0000 UTC m=+2660.997403215" watchObservedRunningTime="2026-10-30 01:23:00.93287599 +0000 UTC m=+2661.004199830"
Oct 30 01:23:01.209243 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:01.209212    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:01.989496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:01.989467    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:23:02.027610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.027582    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-kube-api-access-nlvgs\" (UniqueName: \"kubernetes.io/projected/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-kube-api-access-nlvgs\") pod \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") "
Oct 30 01:23:02.027802 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.027757    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-xks-ca\") pod \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") "
Oct 30 01:23:02.027959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.027942    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-openshift-ca\") pod \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\" (UID: \"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a\") "
Oct 30 01:23:02.028325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.028300    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-openshift-ca" pod "94cd3ef8-840a-4de0-ae7c-407eeaf59a3a" (UID: "94cd3ef8-840a-4de0-ae7c-407eeaf59a3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:02.028636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.028622    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:02.030179 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.030150    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-kube-api-access-nlvgs" pod "94cd3ef8-840a-4de0-ae7c-407eeaf59a3a" (UID: "94cd3ef8-840a-4de0-ae7c-407eeaf59a3a"). InnerVolumeSpecName "kube-api-access-nlvgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:02.030314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.030184    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-xks-ca" pod "94cd3ef8-840a-4de0-ae7c-407eeaf59a3a" (UID: "94cd3ef8-840a-4de0-ae7c-407eeaf59a3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:02.130141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.130107    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvgs\" (UniqueName: \"kubernetes.io/projected/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-kube-api-access-nlvgs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:02.130141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.130138    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:02.825960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.825922    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs" event={"ID":"94cd3ef8-840a-4de0-ae7c-407eeaf59a3a","Type":"ContainerDied","Data":"c00380eccdbdb9411651a64bc571af512dd67c0d1ee3f088e28c37efc00f767a"}
Oct 30 01:23:02.826123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.826002    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c00380eccdbdb9411651a64bc571af512dd67c0d1ee3f088e28c37efc00f767a"
Oct 30 01:23:02.826123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.825936    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"
Oct 30 01:23:02.933908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:02.933854    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:03.031291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:03.031255    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"]
Oct 30 01:23:03.035352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:03.035330    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf7777d-2mf2j"]
Oct 30 01:23:04.109760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:04.109728    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e63a1cd-fe40-4777-97a8-17f7bd021f74" path="/var/lib/kubelet/pods/2e63a1cd-fe40-4777-97a8-17f7bd021f74/volumes"
Oct 30 01:23:05.186997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:05.186949    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:06.006789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:06.006751    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:06.449931 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:06.449773    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:06.724741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:06.724647    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:07.054373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:07.054065    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:07.484865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:07.484828    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:08.122517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:08.122482    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:09.733451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:09.733411    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:11.819579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:11.819546    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:23:11.820029 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:11.819927    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:23:11.853843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:11.853810    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:12.510110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:12.510075    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"]
Oct 30 01:23:12.554153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:12.554124    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:13.031044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:13.031012    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"]
Oct 30 01:23:13.035024 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:13.034995    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c35562-dtz9g"]
Oct 30 01:23:13.122724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:13.122693    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:13.389632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:13.389548    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:13.526116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:13.526081    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:13.653900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:13.653804    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:13.845954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:13.845916    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:13.973511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:13.973479    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:14.275450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.275340    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:14.311742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.311681    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bdfb97-a06f-449e-b603-e61f7cbecefd" path="/var/lib/kubelet/pods/b0bdfb97-a06f-449e-b603-e61f7cbecefd/volumes"
Oct 30 01:23:14.312196 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.312174    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"]
Oct 30 01:23:14.458713 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.458679    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:14.569692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.569610    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"]
Oct 30 01:23:14.573669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.573639    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f461c3-cb6sk"]
Oct 30 01:23:14.576405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.576365    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:14.637023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.636992    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"]
Oct 30 01:23:14.637733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.637715    4576 cpu_manager.go:414] "containerMap: removing container" podUID="94cd3ef8-840a-4de0-ae7c-407eeaf59a3a" containerName="revoke-keys"
Oct 30 01:23:14.637733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.637734    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94cd3ef8-840a-4de0-ae7c-407eeaf59a3a" containerName="revoke-keys"
Oct 30 01:23:14.637871 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.637863    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="94cd3ef8-840a-4de0-ae7c-407eeaf59a3a" containerName="revoke-keys"
Oct 30 01:23:14.642448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.642420    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.666314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.659884    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:23:14.666314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.660197    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:14.666314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.663976    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"]
Oct 30 01:23:14.721079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.721034    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:14.727168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.727132    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.727374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.727351    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6js\" (UniqueName: \"kubernetes.io/projected/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-kube-api-access-2k6js\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.727439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.727419    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.829490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.829379    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6js\" (UniqueName: \"kubernetes.io/projected/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-kube-api-access-2k6js\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.829673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.829509    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.829842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.829821    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.830551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.830512    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.832271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.832248    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.851426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.851396    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6js\" (UniqueName: \"kubernetes.io/projected/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-kube-api-access-2k6js\") pod \"maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:14.966093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:14.966058    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:15.051263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:15.050683    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:15.280911 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:15.280858    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbca4a7b_9ce6_4433_b883_9d7ada288cf3.slice/crio-85be914f48c22a2267385d21fb7029e6a9928dbd7265fb6ec3d84d7564aafb09 WatchSource:0}: Error finding container 85be914f48c22a2267385d21fb7029e6a9928dbd7265fb6ec3d84d7564aafb09: Status 404 returned error can't find the container with id 85be914f48c22a2267385d21fb7029e6a9928dbd7265fb6ec3d84d7564aafb09
Oct 30 01:23:15.305799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:15.305762    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"]
Oct 30 01:23:16.109778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:16.109741    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c7e342-20f8-4827-9d14-9b4a01d65e52" path="/var/lib/kubelet/pods/66c7e342-20f8-4827-9d14-9b4a01d65e52/volumes"
Oct 30 01:23:16.252551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:16.252512    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg" event={"ID":"bbca4a7b-9ce6-4433-b883-9d7ada288cf3","Type":"ContainerStarted","Data":"0ee960fed50b799b96b9d070ea1db267c7ef8c1a900348b56ac61126f0a5a381"}
Oct 30 01:23:16.252725 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:16.252576    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg" event={"ID":"bbca4a7b-9ce6-4433-b883-9d7ada288cf3","Type":"ContainerStarted","Data":"85be914f48c22a2267385d21fb7029e6a9928dbd7265fb6ec3d84d7564aafb09"}
Oct 30 01:23:16.296141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:16.296086    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg" podStartSLOduration=2.296071098 podStartE2EDuration="2.296071098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:16.293759611 +0000 UTC m=+2676.365083445" watchObservedRunningTime="2026-10-30 01:23:16.296071098 +0000 UTC m=+2676.367394932"
Oct 30 01:23:16.768937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:16.768900    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:17.314199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:17.314158    4576 generic.go:356] "Generic (PLEG): container finished" podID="bbca4a7b-9ce6-4433-b883-9d7ada288cf3" containerID="0ee960fed50b799b96b9d070ea1db267c7ef8c1a900348b56ac61126f0a5a381" exitCode=0
Oct 30 01:23:17.314821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:17.314794    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg" event={"ID":"bbca4a7b-9ce6-4433-b883-9d7ada288cf3","Type":"ContainerDied","Data":"0ee960fed50b799b96b9d070ea1db267c7ef8c1a900348b56ac61126f0a5a381"}
Oct 30 01:23:18.220882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.220851    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"]
Oct 30 01:23:18.226048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.226028    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.239723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.239696    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"]
Oct 30 01:23:18.245439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.245415    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.249615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.249588    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"]
Oct 30 01:23:18.269139 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.269108    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"]
Oct 30 01:23:18.368923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.368884    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcnmx\" (UniqueName: \"kubernetes.io/projected/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-kube-api-access-fcnmx\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.369503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.369018    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjbm\" (UniqueName: \"kubernetes.io/projected/b000e779-6421-4220-bd3e-5240b8b7bdea-kube-api-access-bjjbm\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.369503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.369061    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.369503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.369273    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.369503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.369318    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.369503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.369435    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.397481 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:18.397444    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:23:18.474328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.472895    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjbm\" (UniqueName: \"kubernetes.io/projected/b000e779-6421-4220-bd3e-5240b8b7bdea-kube-api-access-bjjbm\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.474328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.472945    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.474328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.473127    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.474328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.473153    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.474328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.473266    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.474328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.473362    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcnmx\" (UniqueName: \"kubernetes.io/projected/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-kube-api-access-fcnmx\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.476493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.476430    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.478505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.478464    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.501354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.490027    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.501354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.499894    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcnmx\" (UniqueName: \"kubernetes.io/projected/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-kube-api-access-fcnmx\") pod \"maas-api-revoke-sub-ba415567-nrltr\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.507382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.503177    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjbm\" (UniqueName: \"kubernetes.io/projected/b000e779-6421-4220-bd3e-5240b8b7bdea-kube-api-access-bjjbm\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.507382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.504142    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76de7d81-fh7v5\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.562018 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.561981    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:18.564458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.564437    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:18.566914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.566892    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:18.581180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.581154    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:18.678159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.676256    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-openshift-ca\") pod \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") "
Oct 30 01:23:18.678159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.676405    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-xks-ca\") pod \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") "
Oct 30 01:23:18.678159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.676489    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-kube-api-access-2k6js\" (UniqueName: \"kubernetes.io/projected/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-kube-api-access-2k6js\") pod \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\" (UID: \"bbca4a7b-9ce6-4433-b883-9d7ada288cf3\") "
Oct 30 01:23:18.678159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.677819    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-openshift-ca" pod "bbca4a7b-9ce6-4433-b883-9d7ada288cf3" (UID: "bbca4a7b-9ce6-4433-b883-9d7ada288cf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:18.689906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.689177    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-kube-api-access-2k6js" pod "bbca4a7b-9ce6-4433-b883-9d7ada288cf3" (UID: "bbca4a7b-9ce6-4433-b883-9d7ada288cf3"). InnerVolumeSpecName "kube-api-access-2k6js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:18.693345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.690879    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-xks-ca" pod "bbca4a7b-9ce6-4433-b883-9d7ada288cf3" (UID: "bbca4a7b-9ce6-4433-b883-9d7ada288cf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:18.778744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.778661    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k6js\" (UniqueName: \"kubernetes.io/projected/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-kube-api-access-2k6js\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:18.778744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.778690    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:18.778744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.778701    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca4a7b-9ce6-4433-b883-9d7ada288cf3-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:18.943208 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:18.943171    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2f3655_2cf9_4489_b5a8_7f33b5ce7a8e.slice/crio-ef5e86ac737606f7e9bc7618bbb111beda4f59293814e0c4e137a7d7e3d42bdc WatchSource:0}: Error finding container ef5e86ac737606f7e9bc7618bbb111beda4f59293814e0c4e137a7d7e3d42bdc: Status 404 returned error can't find the container with id ef5e86ac737606f7e9bc7618bbb111beda4f59293814e0c4e137a7d7e3d42bdc
Oct 30 01:23:18.943208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.943194    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"]
Oct 30 01:23:18.954969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:18.954942    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"]
Oct 30 01:23:18.957065 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:18.957037    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb000e779_6421_4220_bd3e_5240b8b7bdea.slice/crio-b7c57473d84f9ea3c2c81c2f0ef0d068cf84e2b86823d076c1b485666b6ac550 WatchSource:0}: Error finding container b7c57473d84f9ea3c2c81c2f0ef0d068cf84e2b86823d076c1b485666b6ac550: Status 404 returned error can't find the container with id b7c57473d84f9ea3c2c81c2f0ef0d068cf84e2b86823d076c1b485666b6ac550
Oct 30 01:23:19.312796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.312753    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d" podUID="5fd49eaf-15c9-4507-9e2b-caa4147544af" containerName="payload-pre-processing" containerID="cri-o://ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39" gracePeriod=25
Oct 30 01:23:19.357632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.357598    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5" event={"ID":"b000e779-6421-4220-bd3e-5240b8b7bdea","Type":"ContainerStarted","Data":"b7c57473d84f9ea3c2c81c2f0ef0d068cf84e2b86823d076c1b485666b6ac550"}
Oct 30 01:23:19.359162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.359138    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"
Oct 30 01:23:19.359366 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.359142    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg" event={"ID":"bbca4a7b-9ce6-4433-b883-9d7ada288cf3","Type":"ContainerDied","Data":"85be914f48c22a2267385d21fb7029e6a9928dbd7265fb6ec3d84d7564aafb09"}
Oct 30 01:23:19.359366 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.359219    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85be914f48c22a2267385d21fb7029e6a9928dbd7265fb6ec3d84d7564aafb09"
Oct 30 01:23:19.360990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.360959    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr" event={"ID":"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e","Type":"ContainerStarted","Data":"656a4e5aaf728fff00a7562c7e54d58d276f3cbd085341d70a569e9eb825f53a"}
Oct 30 01:23:19.361141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.361010    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr" event={"ID":"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e","Type":"ContainerStarted","Data":"ef5e86ac737606f7e9bc7618bbb111beda4f59293814e0c4e137a7d7e3d42bdc"}
Oct 30 01:23:19.404288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.404221    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr" podStartSLOduration=1.404207222 podStartE2EDuration="1.404207222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:19.402682831 +0000 UTC m=+2679.474006691" watchObservedRunningTime="2026-10-30 01:23:19.404207222 +0000 UTC m=+2679.475531056"
Oct 30 01:23:19.653213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.653183    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:23:19.691852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.691819    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd49eaf-15c9-4507-9e2b-caa4147544af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd49eaf-15c9-4507-9e2b-caa4147544af-plugins-config-volume\") pod \"5fd49eaf-15c9-4507-9e2b-caa4147544af\" (UID: \"5fd49eaf-15c9-4507-9e2b-caa4147544af\") "
Oct 30 01:23:19.692151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.692131    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd49eaf-15c9-4507-9e2b-caa4147544af-kube-api-access-kfh2s\" (UniqueName: \"kubernetes.io/projected/5fd49eaf-15c9-4507-9e2b-caa4147544af-kube-api-access-kfh2s\") pod \"5fd49eaf-15c9-4507-9e2b-caa4147544af\" (UID: \"5fd49eaf-15c9-4507-9e2b-caa4147544af\") "
Oct 30 01:23:19.692262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.692224    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd49eaf-15c9-4507-9e2b-caa4147544af-plugins-config-volume" pod "5fd49eaf-15c9-4507-9e2b-caa4147544af" (UID: "5fd49eaf-15c9-4507-9e2b-caa4147544af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:19.692750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.692734    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fd49eaf-15c9-4507-9e2b-caa4147544af-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:19.694390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.694367    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd49eaf-15c9-4507-9e2b-caa4147544af-kube-api-access-kfh2s" pod "5fd49eaf-15c9-4507-9e2b-caa4147544af" (UID: "5fd49eaf-15c9-4507-9e2b-caa4147544af"). InnerVolumeSpecName "kube-api-access-kfh2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:19.794524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.794451    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfh2s\" (UniqueName: \"kubernetes.io/projected/5fd49eaf-15c9-4507-9e2b-caa4147544af-kube-api-access-kfh2s\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:19.992704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.992666    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"]
Oct 30 01:23:19.994021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.993852    4576 cpu_manager.go:414] "containerMap: removing container" podUID="bbca4a7b-9ce6-4433-b883-9d7ada288cf3" containerName="revoke-keys"
Oct 30 01:23:19.994021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.993875    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbca4a7b-9ce6-4433-b883-9d7ada288cf3" containerName="revoke-keys"
Oct 30 01:23:19.994021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.993928    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5fd49eaf-15c9-4507-9e2b-caa4147544af" containerName="payload-pre-processing"
Oct 30 01:23:19.994021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.993936    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd49eaf-15c9-4507-9e2b-caa4147544af" containerName="payload-pre-processing"
Oct 30 01:23:19.995104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.994191    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbca4a7b-9ce6-4433-b883-9d7ada288cf3" containerName="revoke-keys"
Oct 30 01:23:19.995104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.994215    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd49eaf-15c9-4507-9e2b-caa4147544af" containerName="payload-pre-processing"
Oct 30 01:23:19.998455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:19.998432    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.012523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.011786    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:20.019991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.019171    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"]
Oct 30 01:23:20.104469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.104431    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtql\" (UniqueName: \"kubernetes.io/projected/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-kube-api-access-jjtql\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.104651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.104494    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.104651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.104636    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.206610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.206571    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtql\" (UniqueName: \"kubernetes.io/projected/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-kube-api-access-jjtql\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.206813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.206654    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.206813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.206767    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.208877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.208850    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.216260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.213822    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.216413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.216328    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:20.233345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.233282    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtql\" (UniqueName: \"kubernetes.io/projected/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-kube-api-access-jjtql\") pod \"maas-api-revoke-sub-43e98a65-qlzkt\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.338411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.338321    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:20.367523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.367456    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:20.375916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.375884    4576 generic.go:356] "Generic (PLEG): container finished" podID="5fd49eaf-15c9-4507-9e2b-caa4147544af" containerID="ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39" exitCode=0
Oct 30 01:23:20.376065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.376012    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"
Oct 30 01:23:20.376065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.376037    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d" event={"ID":"5fd49eaf-15c9-4507-9e2b-caa4147544af","Type":"ContainerDied","Data":"ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39"}
Oct 30 01:23:20.376205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.376085    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d" event={"ID":"5fd49eaf-15c9-4507-9e2b-caa4147544af","Type":"ContainerDied","Data":"c3e3f9d030353f2580662820b03a7ea3f00b60ba52668f1b3727172d93046dff"}
Oct 30 01:23:20.376205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.376109    4576 scope.go:122] "RemoveContainer" containerID="ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39"
Oct 30 01:23:20.380549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.380521    4576 generic.go:356] "Generic (PLEG): container finished" podID="b000e779-6421-4220-bd3e-5240b8b7bdea" containerID="4600f8d172cfa2bb23d9a9335df392c1b379dbd59a586846554d881bb170e9e1" exitCode=0
Oct 30 01:23:20.380671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.380607    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5" event={"ID":"b000e779-6421-4220-bd3e-5240b8b7bdea","Type":"ContainerDied","Data":"4600f8d172cfa2bb23d9a9335df392c1b379dbd59a586846554d881bb170e9e1"}
Oct 30 01:23:20.388224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.385485    4576 generic.go:356] "Generic (PLEG): container finished" podID="1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e" containerID="656a4e5aaf728fff00a7562c7e54d58d276f3cbd085341d70a569e9eb825f53a" exitCode=0
Oct 30 01:23:20.388224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.385748    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr" event={"ID":"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e","Type":"ContainerDied","Data":"656a4e5aaf728fff00a7562c7e54d58d276f3cbd085341d70a569e9eb825f53a"}
Oct 30 01:23:20.394590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.394573    4576 scope.go:122] "RemoveContainer" containerID="ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39"
Oct 30 01:23:20.394907 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:20.394884    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39\": container with ID starting with ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39 not found: ID does not exist" containerID="ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39"
Oct 30 01:23:20.394966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.394916    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39"} err="failed to get container status \"ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39\": rpc error: code = NotFound desc = could not find container \"ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39\": container with ID starting with ff1574bba986d4ad7fb651cf03e05f30096493d009d6397c92a2c312ab9cfa39 not found: ID does not exist"
Oct 30 01:23:20.440409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.440370    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"]
Oct 30 01:23:20.507575 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.507544    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96bd13-7bc786cc47-mgk9d"]
Oct 30 01:23:20.722255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.721868    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:20.794210 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:20.794178    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68371e2a_9cb2_48ae_bdc3_6d5e055ef91d.slice/crio-c887ee8759efd0ee5c28c64e9dbdb9b58486826ff8306f697e93c854ad4d7668 WatchSource:0}: Error finding container c887ee8759efd0ee5c28c64e9dbdb9b58486826ff8306f697e93c854ad4d7668: Status 404 returned error can't find the container with id c887ee8759efd0ee5c28c64e9dbdb9b58486826ff8306f697e93c854ad4d7668
Oct 30 01:23:20.803433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.803403    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"]
Oct 30 01:23:20.860304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.860275    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"]
Oct 30 01:23:20.864405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.864376    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1075cb-llmrb"]
Oct 30 01:23:20.871819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.871783    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:20.995702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:20.995593    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:21.333542 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.333451    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:21.397069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.397033    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt" event={"ID":"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d","Type":"ContainerStarted","Data":"1419364690e6c194a5f33a8d57b3ec4d9f072ddaf785ca206b5405292cf9cbb5"}
Oct 30 01:23:21.397299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.397095    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt" event={"ID":"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d","Type":"ContainerStarted","Data":"c887ee8759efd0ee5c28c64e9dbdb9b58486826ff8306f697e93c854ad4d7668"}
Oct 30 01:23:21.441505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.441451    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt" podStartSLOduration=2.441436797 podStartE2EDuration="2.441436797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:21.440125861 +0000 UTC m=+2681.511449698" watchObservedRunningTime="2026-10-30 01:23:21.441436797 +0000 UTC m=+2681.512760632"
Oct 30 01:23:21.609252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.608388    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:21.682115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.682085    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:21.718480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.718452    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b000e779-6421-4220-bd3e-5240b8b7bdea-kube-api-access-bjjbm\" (UniqueName: \"kubernetes.io/projected/b000e779-6421-4220-bd3e-5240b8b7bdea-kube-api-access-bjjbm\") pod \"b000e779-6421-4220-bd3e-5240b8b7bdea\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") "
Oct 30 01:23:21.718665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.718557    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-openshift-ca\") pod \"b000e779-6421-4220-bd3e-5240b8b7bdea\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") "
Oct 30 01:23:21.718665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.718587    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-xks-ca\") pod \"b000e779-6421-4220-bd3e-5240b8b7bdea\" (UID: \"b000e779-6421-4220-bd3e-5240b8b7bdea\") "
Oct 30 01:23:21.719003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.718973    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-openshift-ca" pod "b000e779-6421-4220-bd3e-5240b8b7bdea" (UID: "b000e779-6421-4220-bd3e-5240b8b7bdea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:21.719498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.719472    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.721296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.721181    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b000e779-6421-4220-bd3e-5240b8b7bdea-kube-api-access-bjjbm" pod "b000e779-6421-4220-bd3e-5240b8b7bdea" (UID: "b000e779-6421-4220-bd3e-5240b8b7bdea"). InnerVolumeSpecName "kube-api-access-bjjbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:21.721472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.721446    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-xks-ca" pod "b000e779-6421-4220-bd3e-5240b8b7bdea" (UID: "b000e779-6421-4220-bd3e-5240b8b7bdea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:21.820608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.820577    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-openshift-ca\") pod \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") "
Oct 30 01:23:21.820795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.820630    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-xks-ca\") pod \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") "
Oct 30 01:23:21.820795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.820782    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-kube-api-access-fcnmx\" (UniqueName: \"kubernetes.io/projected/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-kube-api-access-fcnmx\") pod \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\" (UID: \"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e\") "
Oct 30 01:23:21.820951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.820921    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-openshift-ca" pod "1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e" (UID: "1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:21.821405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.821376    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.821528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.821405    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjjbm\" (UniqueName: \"kubernetes.io/projected/b000e779-6421-4220-bd3e-5240b8b7bdea-kube-api-access-bjjbm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.821528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.821420    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b000e779-6421-4220-bd3e-5240b8b7bdea-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.822877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.822849    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-xks-ca" pod "1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e" (UID: "1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:21.823479 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.823460    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-kube-api-access-fcnmx" pod "1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e" (UID: "1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e"). InnerVolumeSpecName "kube-api-access-fcnmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:21.878022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.877916    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:21.923901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.923865    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.923901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:21.923904    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcnmx\" (UniqueName: \"kubernetes.io/projected/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e-kube-api-access-fcnmx\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:22.028727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.028679    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:22.118796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.118758    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd49eaf-15c9-4507-9e2b-caa4147544af" path="/var/lib/kubelet/pods/5fd49eaf-15c9-4507-9e2b-caa4147544af/volumes"
Oct 30 01:23:22.119554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.119521    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf4c8b5-04b6-4346-b2d8-20f144b6853f" path="/var/lib/kubelet/pods/7cf4c8b5-04b6-4346-b2d8-20f144b6853f/volumes"
Oct 30 01:23:22.196753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.196714    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:22.412941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.412858    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"
Oct 30 01:23:22.412941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.412890    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5" event={"ID":"b000e779-6421-4220-bd3e-5240b8b7bdea","Type":"ContainerDied","Data":"b7c57473d84f9ea3c2c81c2f0ef0d068cf84e2b86823d076c1b485666b6ac550"}
Oct 30 01:23:22.413185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.412951    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7c57473d84f9ea3c2c81c2f0ef0d068cf84e2b86823d076c1b485666b6ac550"
Oct 30 01:23:22.415019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.414991    4576 generic.go:356] "Generic (PLEG): container finished" podID="68371e2a-9cb2-48ae-bdc3-6d5e055ef91d" containerID="1419364690e6c194a5f33a8d57b3ec4d9f072ddaf785ca206b5405292cf9cbb5" exitCode=0
Oct 30 01:23:22.415167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.415079    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt" event={"ID":"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d","Type":"ContainerDied","Data":"1419364690e6c194a5f33a8d57b3ec4d9f072ddaf785ca206b5405292cf9cbb5"}
Oct 30 01:23:22.418843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.418821    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"
Oct 30 01:23:22.418973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.418890    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr" event={"ID":"1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e","Type":"ContainerDied","Data":"ef5e86ac737606f7e9bc7618bbb111beda4f59293814e0c4e137a7d7e3d42bdc"}
Oct 30 01:23:22.418973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.418951    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef5e86ac737606f7e9bc7618bbb111beda4f59293814e0c4e137a7d7e3d42bdc"
Oct 30 01:23:22.668681 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.668589    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"]
Oct 30 01:23:22.669592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.669569    4576 cpu_manager.go:414] "containerMap: removing container" podUID="1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e" containerName="revoke-keys"
Oct 30 01:23:22.669592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.669591    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e" containerName="revoke-keys"
Oct 30 01:23:22.669764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.669657    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b000e779-6421-4220-bd3e-5240b8b7bdea" containerName="revoke-keys"
Oct 30 01:23:22.669764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.669666    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b000e779-6421-4220-bd3e-5240b8b7bdea" containerName="revoke-keys"
Oct 30 01:23:22.669877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.669836    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b000e779-6421-4220-bd3e-5240b8b7bdea" containerName="revoke-keys"
Oct 30 01:23:22.669877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.669862    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e" containerName="revoke-keys"
Oct 30 01:23:22.674917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.674895    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.692472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.692445    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"]
Oct 30 01:23:22.742084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.742051    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.742292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.742260    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.742468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.742438    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxdc\" (UniqueName: \"kubernetes.io/projected/89f507e4-f213-4b00-b2c7-ea162731c3d1-kube-api-access-4bxdc\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.844571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.844540    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.844754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.844624    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.844754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.844721    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxdc\" (UniqueName: \"kubernetes.io/projected/89f507e4-f213-4b00-b2c7-ea162731c3d1-kube-api-access-4bxdc\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.845346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.845324    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.847205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.847183    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.865632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.865596    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxdc\" (UniqueName: \"kubernetes.io/projected/89f507e4-f213-4b00-b2c7-ea162731c3d1-kube-api-access-4bxdc\") pod \"maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:22.990089 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:22.990053    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:23.117644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.117609    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:23.298004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.297763    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"]
Oct 30 01:23:23.298004 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:23.297893    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f507e4_f213_4b00_b2c7_ea162731c3d1.slice/crio-3e9b65abdf8dd3f9adb7a0f079ad2b9cce6c7a3529bd0d1aff1ddb9bb3b00f88 WatchSource:0}: Error finding container 3e9b65abdf8dd3f9adb7a0f079ad2b9cce6c7a3529bd0d1aff1ddb9bb3b00f88: Status 404 returned error can't find the container with id 3e9b65abdf8dd3f9adb7a0f079ad2b9cce6c7a3529bd0d1aff1ddb9bb3b00f88
Oct 30 01:23:23.428203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.428162    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt" event={"ID":"89f507e4-f213-4b00-b2c7-ea162731c3d1","Type":"ContainerStarted","Data":"3e9b65abdf8dd3f9adb7a0f079ad2b9cce6c7a3529bd0d1aff1ddb9bb3b00f88"}
Oct 30 01:23:23.635083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.635051    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:23.762783 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.762751    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-xks-ca\") pod \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") "
Oct 30 01:23:23.763185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.762940    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-openshift-ca\") pod \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") "
Oct 30 01:23:23.763185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.763094    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-kube-api-access-jjtql\" (UniqueName: \"kubernetes.io/projected/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-kube-api-access-jjtql\") pod \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\" (UID: \"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d\") "
Oct 30 01:23:23.763399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.763378    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-openshift-ca" pod "68371e2a-9cb2-48ae-bdc3-6d5e055ef91d" (UID: "68371e2a-9cb2-48ae-bdc3-6d5e055ef91d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:23.763771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.763751    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.767878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.767848    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-xks-ca" pod "68371e2a-9cb2-48ae-bdc3-6d5e055ef91d" (UID: "68371e2a-9cb2-48ae-bdc3-6d5e055ef91d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:23.768223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.768198    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-kube-api-access-jjtql" pod "68371e2a-9cb2-48ae-bdc3-6d5e055ef91d" (UID: "68371e2a-9cb2-48ae-bdc3-6d5e055ef91d"). InnerVolumeSpecName "kube-api-access-jjtql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:23.864896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.864805    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtql\" (UniqueName: \"kubernetes.io/projected/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-kube-api-access-jjtql\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.864896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:23.864848    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:24.438643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:24.438611    4576 generic.go:356] "Generic (PLEG): container finished" podID="89f507e4-f213-4b00-b2c7-ea162731c3d1" containerID="fb4090cfdabb7f67f442ce4f4c17e67b92e079704327c418366fa236e2bce6b3" exitCode=0
Oct 30 01:23:24.438821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:24.438695    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt" event={"ID":"89f507e4-f213-4b00-b2c7-ea162731c3d1","Type":"ContainerDied","Data":"fb4090cfdabb7f67f442ce4f4c17e67b92e079704327c418366fa236e2bce6b3"}
Oct 30 01:23:24.440279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:24.440257    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt" event={"ID":"68371e2a-9cb2-48ae-bdc3-6d5e055ef91d","Type":"ContainerDied","Data":"c887ee8759efd0ee5c28c64e9dbdb9b58486826ff8306f697e93c854ad4d7668"}
Oct 30 01:23:24.440397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:24.440283    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"
Oct 30 01:23:24.440397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:24.440299    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c887ee8759efd0ee5c28c64e9dbdb9b58486826ff8306f697e93c854ad4d7668"
Oct 30 01:23:24.969840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:24.969798    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:25.632363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.632338    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:25.782712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.782596    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"]
Oct 30 01:23:25.798073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.798035    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f507e4-f213-4b00-b2c7-ea162731c3d1-kube-api-access-4bxdc\" (UniqueName: \"kubernetes.io/projected/89f507e4-f213-4b00-b2c7-ea162731c3d1-kube-api-access-4bxdc\") pod \"89f507e4-f213-4b00-b2c7-ea162731c3d1\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") "
Oct 30 01:23:25.798286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.798128    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-openshift-ca\") pod \"89f507e4-f213-4b00-b2c7-ea162731c3d1\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") "
Oct 30 01:23:25.798286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.798199    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-xks-ca\") pod \"89f507e4-f213-4b00-b2c7-ea162731c3d1\" (UID: \"89f507e4-f213-4b00-b2c7-ea162731c3d1\") "
Oct 30 01:23:25.798988 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.798958    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-openshift-ca" pod "89f507e4-f213-4b00-b2c7-ea162731c3d1" (UID: "89f507e4-f213-4b00-b2c7-ea162731c3d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:25.801149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.801126    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f507e4-f213-4b00-b2c7-ea162731c3d1-kube-api-access-4bxdc" pod "89f507e4-f213-4b00-b2c7-ea162731c3d1" (UID: "89f507e4-f213-4b00-b2c7-ea162731c3d1"). InnerVolumeSpecName "kube-api-access-4bxdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:25.804283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.803550    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-xks-ca" pod "89f507e4-f213-4b00-b2c7-ea162731c3d1" (UID: "89f507e4-f213-4b00-b2c7-ea162731c3d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:25.820406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.818153    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:25.902333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.900808    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxdc\" (UniqueName: \"kubernetes.io/projected/89f507e4-f213-4b00-b2c7-ea162731c3d1-kube-api-access-4bxdc\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:25.902333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.900845    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:25.902333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.900863    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f507e4-f213-4b00-b2c7-ea162731c3d1-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:25.976296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.972951    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"]
Oct 30 01:23:25.976296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.974080    4576 cpu_manager.go:414] "containerMap: removing container" podUID="89f507e4-f213-4b00-b2c7-ea162731c3d1" containerName="revoke-keys"
Oct 30 01:23:25.976296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.974103    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f507e4-f213-4b00-b2c7-ea162731c3d1" containerName="revoke-keys"
Oct 30 01:23:25.976296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.974136    4576 cpu_manager.go:414] "containerMap: removing container" podUID="68371e2a-9cb2-48ae-bdc3-6d5e055ef91d" containerName="revoke-keys"
Oct 30 01:23:25.976296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.974147    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68371e2a-9cb2-48ae-bdc3-6d5e055ef91d" containerName="revoke-keys"
Oct 30 01:23:25.976296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.974442    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f507e4-f213-4b00-b2c7-ea162731c3d1" containerName="revoke-keys"
Oct 30 01:23:25.976296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.974463    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="68371e2a-9cb2-48ae-bdc3-6d5e055ef91d" containerName="revoke-keys"
Oct 30 01:23:25.982297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:25.982269    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.013257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.009191    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"]
Oct 30 01:23:26.108245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.108138    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.108245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.108212    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqrn4\" (UniqueName: \"kubernetes.io/projected/e4d29976-e999-42de-865e-5b31caaa9032-kube-api-access-kqrn4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.108457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.108338    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.209646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.209612    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.209808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.209665    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqrn4\" (UniqueName: \"kubernetes.io/projected/e4d29976-e999-42de-865e-5b31caaa9032-kube-api-access-kqrn4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.209808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.209774    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.210585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.210560    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.212295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.212271    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.233317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.233289    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqrn4\" (UniqueName: \"kubernetes.io/projected/e4d29976-e999-42de-865e-5b31caaa9032-kube-api-access-kqrn4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.319079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.319045    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:26.459197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.459158    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt" event={"ID":"89f507e4-f213-4b00-b2c7-ea162731c3d1","Type":"ContainerDied","Data":"3e9b65abdf8dd3f9adb7a0f079ad2b9cce6c7a3529bd0d1aff1ddb9bb3b00f88"}
Oct 30 01:23:26.459197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.459187    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"
Oct 30 01:23:26.459471 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.459224    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9b65abdf8dd3f9adb7a0f079ad2b9cce6c7a3529bd0d1aff1ddb9bb3b00f88"
Oct 30 01:23:26.625561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.625391    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"]
Oct 30 01:23:26.628920 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:26.628817    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d29976_e999_42de_865e_5b31caaa9032.slice/crio-d7bfcf6eb50ee74edfbd77f5be10bc0461887aa89637ea75f1c6a4bb3395c280 WatchSource:0}: Error finding container d7bfcf6eb50ee74edfbd77f5be10bc0461887aa89637ea75f1c6a4bb3395c280: Status 404 returned error can't find the container with id d7bfcf6eb50ee74edfbd77f5be10bc0461887aa89637ea75f1c6a4bb3395c280
Oct 30 01:23:26.882066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:26.881966    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:27.422836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:27.422799    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:27.495303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:27.492890    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf" event={"ID":"e4d29976-e999-42de-865e-5b31caaa9032","Type":"ContainerStarted","Data":"c1c72234702f0df3a7b4435d9fd618be8210882b2c5dabf99b472dd0c0e6a218"}
Oct 30 01:23:27.495303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:27.493007    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf" event={"ID":"e4d29976-e999-42de-865e-5b31caaa9032","Type":"ContainerStarted","Data":"d7bfcf6eb50ee74edfbd77f5be10bc0461887aa89637ea75f1c6a4bb3395c280"}
Oct 30 01:23:27.538633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:27.538595    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main" containerID="cri-o://90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0" gracePeriod=45
Oct 30 01:23:27.538963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:27.538894    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf" podStartSLOduration=2.538881083 podStartE2EDuration="2.538881083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:27.537729737 +0000 UTC m=+2687.609053570" watchObservedRunningTime="2026-10-30 01:23:27.538881083 +0000 UTC m=+2687.610204918"
Oct 30 01:23:27.541299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:27.541189    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:27.752510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:27.752474    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:28.099800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.099777    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:23:28.152386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.152351    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tmp-dir\") pod \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") "
Oct 30 01:23:28.152557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.152432    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-model-cache\") pod \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") "
Oct 30 01:23:28.152557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.152463    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-kube-api-access-kcflw\" (UniqueName: \"kubernetes.io/projected/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-kube-api-access-kcflw\") pod \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") "
Oct 30 01:23:28.152557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.152508    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-dshm\") pod \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") "
Oct 30 01:23:28.152690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.152577    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-home\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-home\") pod \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") "
Oct 30 01:23:28.152739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.152725    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tls-certs\") pod \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\" (UID: \"5f2f69c0-1ec3-460d-b8a4-c79f166a9870\") "
Oct 30 01:23:28.153886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.153855    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tmp-dir" pod "5f2f69c0-1ec3-460d-b8a4-c79f166a9870" (UID: "5f2f69c0-1ec3-460d-b8a4-c79f166a9870"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:28.154075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.154043    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-model-cache" pod "5f2f69c0-1ec3-460d-b8a4-c79f166a9870" (UID: "5f2f69c0-1ec3-460d-b8a4-c79f166a9870"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:28.155040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.154719    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-home" pod "5f2f69c0-1ec3-460d-b8a4-c79f166a9870" (UID: "5f2f69c0-1ec3-460d-b8a4-c79f166a9870"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:28.157439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.157406    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-kube-api-access-kcflw" pod "5f2f69c0-1ec3-460d-b8a4-c79f166a9870" (UID: "5f2f69c0-1ec3-460d-b8a4-c79f166a9870"). InnerVolumeSpecName "kube-api-access-kcflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:28.157999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.157975    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-dshm" pod "5f2f69c0-1ec3-460d-b8a4-c79f166a9870" (UID: "5f2f69c0-1ec3-460d-b8a4-c79f166a9870"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:28.158970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.158951    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tls-certs" pod "5f2f69c0-1ec3-460d-b8a4-c79f166a9870" (UID: "5f2f69c0-1ec3-460d-b8a4-c79f166a9870"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:28.200657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.200624    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"]
Oct 30 01:23:28.201631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.201592    4576 cpu_manager.go:414] "containerMap: removing container" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main"
Oct 30 01:23:28.201631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.201618    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main"
Oct 30 01:23:28.201854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.201795    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main"
Oct 30 01:23:28.205847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.205823    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.222915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.222888    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"]
Oct 30 01:23:28.255118 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255080    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.255305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255142    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.255488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255459    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgb5j\" (UniqueName: \"kubernetes.io/projected/ba1e782b-1121-4da0-a74d-18b8f624136a-kube-api-access-wgb5j\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.255711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255696    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:28.255751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255715    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:28.255751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255724    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:28.255751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255732    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcflw\" (UniqueName: \"kubernetes.io/projected/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-kube-api-access-kcflw\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:28.255751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255741    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:28.255751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.255749    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f2f69c0-1ec3-460d-b8a4-c79f166a9870-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:28.357100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.357013    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.357100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.357065    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.357341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.357175    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgb5j\" (UniqueName: \"kubernetes.io/projected/ba1e782b-1121-4da0-a74d-18b8f624136a-kube-api-access-wgb5j\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.357812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.357789    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.359866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.359838    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.377970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.377947    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgb5j\" (UniqueName: \"kubernetes.io/projected/ba1e782b-1121-4da0-a74d-18b8f624136a-kube-api-access-wgb5j\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.502325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.502286    4576 generic.go:356] "Generic (PLEG): container finished" podID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerID="90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0" exitCode=0
Oct 30 01:23:28.502787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.502330    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"
Oct 30 01:23:28.502787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.502370    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" event={"ID":"5f2f69c0-1ec3-460d-b8a4-c79f166a9870","Type":"ContainerDied","Data":"90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0"}
Oct 30 01:23:28.502787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.502416    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" event={"ID":"5f2f69c0-1ec3-460d-b8a4-c79f166a9870","Type":"ContainerDied","Data":"2337964c3fd0bbe8f3d3014412f3db7dbeb1ae8a883221a9b4f12ef66b1bf5ae"}
Oct 30 01:23:28.502787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.502433    4576 scope.go:122] "RemoveContainer" containerID="90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0"
Oct 30 01:23:28.504574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.504546    4576 generic.go:356] "Generic (PLEG): container finished" podID="e4d29976-e999-42de-865e-5b31caaa9032" containerID="c1c72234702f0df3a7b4435d9fd618be8210882b2c5dabf99b472dd0c0e6a218" exitCode=0
Oct 30 01:23:28.504700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.504626    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf" event={"ID":"e4d29976-e999-42de-865e-5b31caaa9032","Type":"ContainerDied","Data":"c1c72234702f0df3a7b4435d9fd618be8210882b2c5dabf99b472dd0c0e6a218"}
Oct 30 01:23:28.514492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.514471    4576 scope.go:122] "RemoveContainer" containerID="90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0"
Oct 30 01:23:28.514812 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:28.514790    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0\": container with ID starting with 90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0 not found: ID does not exist" containerID="90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0"
Oct 30 01:23:28.514901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.514825    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0"} err="failed to get container status \"90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0\": rpc error: code = NotFound desc = could not find container \"90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0\": container with ID starting with 90dd5e728d3fc23d9b76ea36495dc7c89fded8e4426b9585d5573421a52aded0 not found: ID does not exist"
Oct 30 01:23:28.524451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.524422    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:28.606961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.606924    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"]
Oct 30 01:23:28.615501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.615423    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5"]
Oct 30 01:23:28.830628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.830594    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"]
Oct 30 01:23:28.830996 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:28.830920    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1e782b_1121_4da0_a74d_18b8f624136a.slice/crio-d0f62340b1b75b7f2fa16e3374956850c13b2d7bf90a94eb7b238585f5058c4c WatchSource:0}: Error finding container d0f62340b1b75b7f2fa16e3374956850c13b2d7bf90a94eb7b238585f5058c4c: Status 404 returned error can't find the container with id d0f62340b1b75b7f2fa16e3374956850c13b2d7bf90a94eb7b238585f5058c4c
Oct 30 01:23:28.851565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.851527    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:28.907354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:28.907296    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-24d5ce/rate-test-model-24d5ce-kserve-565ff54df7-khdt5" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:23:29.515942 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.515898    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx" event={"ID":"ba1e782b-1121-4da0-a74d-18b8f624136a","Type":"ContainerStarted","Data":"1aa5eb21504b108e6bf65be91768f19914d36c23af8da74a7e4398117519815e"}
Oct 30 01:23:29.516382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.515962    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx" event={"ID":"ba1e782b-1121-4da0-a74d-18b8f624136a","Type":"ContainerStarted","Data":"d0f62340b1b75b7f2fa16e3374956850c13b2d7bf90a94eb7b238585f5058c4c"}
Oct 30 01:23:29.562457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.562367    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx" podStartSLOduration=1.562352423 podStartE2EDuration="1.562352423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:29.557031824 +0000 UTC m=+2689.628355660" watchObservedRunningTime="2026-10-30 01:23:29.562352423 +0000 UTC m=+2689.633676257"
Oct 30 01:23:29.690789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.690764    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:29.782103 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.782015    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d29976-e999-42de-865e-5b31caaa9032-kube-api-access-kqrn4\" (UniqueName: \"kubernetes.io/projected/e4d29976-e999-42de-865e-5b31caaa9032-kube-api-access-kqrn4\") pod \"e4d29976-e999-42de-865e-5b31caaa9032\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") "
Oct 30 01:23:29.782326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.782304    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-xks-ca\") pod \"e4d29976-e999-42de-865e-5b31caaa9032\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") "
Oct 30 01:23:29.782484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.782455    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-openshift-ca\") pod \"e4d29976-e999-42de-865e-5b31caaa9032\" (UID: \"e4d29976-e999-42de-865e-5b31caaa9032\") "
Oct 30 01:23:29.783133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.783089    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-openshift-ca" pod "e4d29976-e999-42de-865e-5b31caaa9032" (UID: "e4d29976-e999-42de-865e-5b31caaa9032"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:29.783470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.783452    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.785111 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.785084    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d29976-e999-42de-865e-5b31caaa9032-kube-api-access-kqrn4" pod "e4d29976-e999-42de-865e-5b31caaa9032" (UID: "e4d29976-e999-42de-865e-5b31caaa9032"). InnerVolumeSpecName "kube-api-access-kqrn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:29.785600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.785576    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-xks-ca" pod "e4d29976-e999-42de-865e-5b31caaa9032" (UID: "e4d29976-e999-42de-865e-5b31caaa9032"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:29.885161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.885125    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d29976-e999-42de-865e-5b31caaa9032-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:29.885161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:29.885158    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqrn4\" (UniqueName: \"kubernetes.io/projected/e4d29976-e999-42de-865e-5b31caaa9032-kube-api-access-kqrn4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:30.110783 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.110709    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2f69c0-1ec3-460d-b8a4-c79f166a9870" path="/var/lib/kubelet/pods/5f2f69c0-1ec3-460d-b8a4-c79f166a9870/volumes"
Oct 30 01:23:30.475279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.475221    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"]
Oct 30 01:23:30.475994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.475971    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e4d29976-e999-42de-865e-5b31caaa9032" containerName="revoke-keys"
Oct 30 01:23:30.475994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.475994    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d29976-e999-42de-865e-5b31caaa9032" containerName="revoke-keys"
Oct 30 01:23:30.476217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.476202    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d29976-e999-42de-865e-5b31caaa9032" containerName="revoke-keys"
Oct 30 01:23:30.480922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.480901    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.505656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.503358    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"]
Oct 30 01:23:30.536838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.536798    4576 generic.go:356] "Generic (PLEG): container finished" podID="ba1e782b-1121-4da0-a74d-18b8f624136a" containerID="1aa5eb21504b108e6bf65be91768f19914d36c23af8da74a7e4398117519815e" exitCode=0
Oct 30 01:23:30.537300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.536884    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx" event={"ID":"ba1e782b-1121-4da0-a74d-18b8f624136a","Type":"ContainerDied","Data":"1aa5eb21504b108e6bf65be91768f19914d36c23af8da74a7e4398117519815e"}
Oct 30 01:23:30.538528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.538494    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf" event={"ID":"e4d29976-e999-42de-865e-5b31caaa9032","Type":"ContainerDied","Data":"d7bfcf6eb50ee74edfbd77f5be10bc0461887aa89637ea75f1c6a4bb3395c280"}
Oct 30 01:23:30.538662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.538534    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7bfcf6eb50ee74edfbd77f5be10bc0461887aa89637ea75f1c6a4bb3395c280"
Oct 30 01:23:30.538662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.538545    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"
Oct 30 01:23:30.595713 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.595685    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qchg\" (UniqueName: \"kubernetes.io/projected/b05c2802-6d7e-4e37-b594-87476c9a7083-kube-api-access-2qchg\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.595899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.595879    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.595945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.595928    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.698080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.698044    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.698328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.698121    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.698440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.698382    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qchg\" (UniqueName: \"kubernetes.io/projected/b05c2802-6d7e-4e37-b594-87476c9a7083-kube-api-access-2qchg\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.698789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.698762    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.700688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.700665    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.721722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.721691    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qchg\" (UniqueName: \"kubernetes.io/projected/b05c2802-6d7e-4e37-b594-87476c9a7083-kube-api-access-2qchg\") pod \"maas-api-revoke-sub-475c543a-tzj8h\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:30.786796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.786715    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:30.827056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:30.827015    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:31.161017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.160986    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"]
Oct 30 01:23:31.161447 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:31.161414    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05c2802_6d7e_4e37_b594_87476c9a7083.slice/crio-7f53a1701b5d2de9c83a0d30564326f17cf725ebdeb0aa73fc5e7b2793a24d48 WatchSource:0}: Error finding container 7f53a1701b5d2de9c83a0d30564326f17cf725ebdeb0aa73fc5e7b2793a24d48: Status 404 returned error can't find the container with id 7f53a1701b5d2de9c83a0d30564326f17cf725ebdeb0aa73fc5e7b2793a24d48
Oct 30 01:23:31.549267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.549195    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h" event={"ID":"b05c2802-6d7e-4e37-b594-87476c9a7083","Type":"ContainerStarted","Data":"fc84190ca28c785f408c6f7741de61cc39d526578f169cb5f9f4d5085ecefb23"}
Oct 30 01:23:31.549751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.549298    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h" event={"ID":"b05c2802-6d7e-4e37-b594-87476c9a7083","Type":"ContainerStarted","Data":"7f53a1701b5d2de9c83a0d30564326f17cf725ebdeb0aa73fc5e7b2793a24d48"}
Oct 30 01:23:31.591363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.591310    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h" podStartSLOduration=1.591296164 podStartE2EDuration="1.591296164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:31.590064454 +0000 UTC m=+2691.661388290" watchObservedRunningTime="2026-10-30 01:23:31.591296164 +0000 UTC m=+2691.662619999"
Oct 30 01:23:31.723940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.723914    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:31.815118 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.815028    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-xks-ca\") pod \"ba1e782b-1121-4da0-a74d-18b8f624136a\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") "
Oct 30 01:23:31.815325 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.815200    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-openshift-ca\") pod \"ba1e782b-1121-4da0-a74d-18b8f624136a\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") "
Oct 30 01:23:31.815477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.815455    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1e782b-1121-4da0-a74d-18b8f624136a-kube-api-access-wgb5j\" (UniqueName: \"kubernetes.io/projected/ba1e782b-1121-4da0-a74d-18b8f624136a-kube-api-access-wgb5j\") pod \"ba1e782b-1121-4da0-a74d-18b8f624136a\" (UID: \"ba1e782b-1121-4da0-a74d-18b8f624136a\") "
Oct 30 01:23:31.815765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.815735    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-openshift-ca" pod "ba1e782b-1121-4da0-a74d-18b8f624136a" (UID: "ba1e782b-1121-4da0-a74d-18b8f624136a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:31.816395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.816368    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.822185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.822155    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-xks-ca" pod "ba1e782b-1121-4da0-a74d-18b8f624136a" (UID: "ba1e782b-1121-4da0-a74d-18b8f624136a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:31.822330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.822205    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1e782b-1121-4da0-a74d-18b8f624136a-kube-api-access-wgb5j" pod "ba1e782b-1121-4da0-a74d-18b8f624136a" (UID: "ba1e782b-1121-4da0-a74d-18b8f624136a"). InnerVolumeSpecName "kube-api-access-wgb5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:31.918093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.918057    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgb5j\" (UniqueName: \"kubernetes.io/projected/ba1e782b-1121-4da0-a74d-18b8f624136a-kube-api-access-wgb5j\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.918093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:31.918086    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1e782b-1121-4da0-a74d-18b8f624136a-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:32.530787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:32.530729    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:32.572258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:32.570145    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx" event={"ID":"ba1e782b-1121-4da0-a74d-18b8f624136a","Type":"ContainerDied","Data":"d0f62340b1b75b7f2fa16e3374956850c13b2d7bf90a94eb7b238585f5058c4c"}
Oct 30 01:23:32.572258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:32.570209    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0f62340b1b75b7f2fa16e3374956850c13b2d7bf90a94eb7b238585f5058c4c"
Oct 30 01:23:32.572258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:32.570363    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"
Oct 30 01:23:32.577509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:32.577067    4576 generic.go:356] "Generic (PLEG): container finished" podID="b05c2802-6d7e-4e37-b594-87476c9a7083" containerID="fc84190ca28c785f408c6f7741de61cc39d526578f169cb5f9f4d5085ecefb23" exitCode=0
Oct 30 01:23:32.577509 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:32.577444    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h" event={"ID":"b05c2802-6d7e-4e37-b594-87476c9a7083","Type":"ContainerDied","Data":"fc84190ca28c785f408c6f7741de61cc39d526578f169cb5f9f4d5085ecefb23"}
Oct 30 01:23:33.761477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.761448    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:33.848176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.848086    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05c2802-6d7e-4e37-b594-87476c9a7083-kube-api-access-2qchg\" (UniqueName: \"kubernetes.io/projected/b05c2802-6d7e-4e37-b594-87476c9a7083-kube-api-access-2qchg\") pod \"b05c2802-6d7e-4e37-b594-87476c9a7083\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") "
Oct 30 01:23:33.848176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.848168    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-xks-ca\") pod \"b05c2802-6d7e-4e37-b594-87476c9a7083\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") "
Oct 30 01:23:33.848487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.848205    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-openshift-ca\") pod \"b05c2802-6d7e-4e37-b594-87476c9a7083\" (UID: \"b05c2802-6d7e-4e37-b594-87476c9a7083\") "
Oct 30 01:23:33.848686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.848659    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-openshift-ca" pod "b05c2802-6d7e-4e37-b594-87476c9a7083" (UID: "b05c2802-6d7e-4e37-b594-87476c9a7083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:33.849348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.849195    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:33.850776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.850750    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-xks-ca" pod "b05c2802-6d7e-4e37-b594-87476c9a7083" (UID: "b05c2802-6d7e-4e37-b594-87476c9a7083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:33.851106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.851090    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05c2802-6d7e-4e37-b594-87476c9a7083-kube-api-access-2qchg" pod "b05c2802-6d7e-4e37-b594-87476c9a7083" (UID: "b05c2802-6d7e-4e37-b594-87476c9a7083"). InnerVolumeSpecName "kube-api-access-2qchg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:33.951656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.951613    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qchg\" (UniqueName: \"kubernetes.io/projected/b05c2802-6d7e-4e37-b594-87476c9a7083-kube-api-access-2qchg\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:33.951848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.951673    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05c2802-6d7e-4e37-b594-87476c9a7083-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:33.957005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:33.956931    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:34.151255 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.149126    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:34.258370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.258334    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:34.383358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.383278    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:34.518335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.516091    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:34.596126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.596079    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h" event={"ID":"b05c2802-6d7e-4e37-b594-87476c9a7083","Type":"ContainerDied","Data":"7f53a1701b5d2de9c83a0d30564326f17cf725ebdeb0aa73fc5e7b2793a24d48"}
Oct 30 01:23:34.596373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.596144    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"
Oct 30 01:23:34.596373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.596199    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f53a1701b5d2de9c83a0d30564326f17cf725ebdeb0aa73fc5e7b2793a24d48"
Oct 30 01:23:34.611024 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.610997    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"]
Oct 30 01:23:34.616626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.615874    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-07a3a8-76hxt"]
Oct 30 01:23:34.636603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:34.636506    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:35.003122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:35.002469    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:35.115797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:35.115758    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:35.557638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:35.557603    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:35.630649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:35.630614    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"]
Oct 30 01:23:35.638146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:35.638119    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5c515e-4cffg"]
Oct 30 01:23:35.793454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:35.793422    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:36.085605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:36.085568    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:36.109460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:36.109426    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f507e4-f213-4b00-b2c7-ea162731c3d1" path="/var/lib/kubelet/pods/89f507e4-f213-4b00-b2c7-ea162731c3d1/volumes"
Oct 30 01:23:36.109969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:36.109952    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbca4a7b-9ce6-4433-b883-9d7ada288cf3" path="/var/lib/kubelet/pods/bbca4a7b-9ce6-4433-b883-9d7ada288cf3/volumes"
Oct 30 01:23:36.202512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:36.202475    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:36.590421 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:36.590332    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:36.726819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:36.726781    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:37.118853 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:37.118818    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:37.380307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:37.373635    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:38.630990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:38.630957    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:40.379295 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:40.379225    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:40.500858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:40.500822    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:40.812581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:40.812526    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main" containerID="cri-o://6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886" gracePeriod=45
Oct 30 01:23:40.851017 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:40.850989    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:40.973531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:40.973427    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:41.135342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.135314    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:23:41.159943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.159909    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"]
Oct 30 01:23:41.160195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.160162    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy" containerID="cri-o://e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219" gracePeriod=30
Oct 30 01:23:41.169619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.169587    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"]
Oct 30 01:23:41.170029 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.169983    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy" containerID="cri-o://cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585" gracePeriod=30
Oct 30 01:23:41.184288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.184199    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"]
Oct 30 01:23:41.184465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.184441    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy" containerID="cri-o://56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac" gracePeriod=30
Oct 30 01:23:41.193552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.193493    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.244218    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-tmp-dir\") pod \"10857aa4-9e02-4728-8581-c32e865769a6\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") "
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.244389    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10857aa4-9e02-4728-8581-c32e865769a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/10857aa4-9e02-4728-8581-c32e865769a6-tls-certs\") pod \"10857aa4-9e02-4728-8581-c32e865769a6\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") "
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.244457    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-dshm\") pod \"10857aa4-9e02-4728-8581-c32e865769a6\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") "
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.244509    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-model-cache\") pod \"10857aa4-9e02-4728-8581-c32e865769a6\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") "
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.244537    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10857aa4-9e02-4728-8581-c32e865769a6-kube-api-access-btrgp\" (UniqueName: \"kubernetes.io/projected/10857aa4-9e02-4728-8581-c32e865769a6-kube-api-access-btrgp\") pod \"10857aa4-9e02-4728-8581-c32e865769a6\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") "
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.244606    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-home\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-home\") pod \"10857aa4-9e02-4728-8581-c32e865769a6\" (UID: \"10857aa4-9e02-4728-8581-c32e865769a6\") "
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.246760    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-home" pod "10857aa4-9e02-4728-8581-c32e865769a6" (UID: "10857aa4-9e02-4728-8581-c32e865769a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.247030    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-tmp-dir" pod "10857aa4-9e02-4728-8581-c32e865769a6" (UID: "10857aa4-9e02-4728-8581-c32e865769a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:41.264583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.247703    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-model-cache" pod "10857aa4-9e02-4728-8581-c32e865769a6" (UID: "10857aa4-9e02-4728-8581-c32e865769a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:41.330863 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.330397    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10857aa4-9e02-4728-8581-c32e865769a6-tls-certs" pod "10857aa4-9e02-4728-8581-c32e865769a6" (UID: "10857aa4-9e02-4728-8581-c32e865769a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:41.365626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.354069    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10857aa4-9e02-4728-8581-c32e865769a6-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:41.365626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.354118    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:41.365626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.354133    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:41.365626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.354146    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:41.365626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.359393    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-dshm" pod "10857aa4-9e02-4728-8581-c32e865769a6" (UID: "10857aa4-9e02-4728-8581-c32e865769a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:41.366672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.366619    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10857aa4-9e02-4728-8581-c32e865769a6-kube-api-access-btrgp" pod "10857aa4-9e02-4728-8581-c32e865769a6" (UID: "10857aa4-9e02-4728-8581-c32e865769a6"). InnerVolumeSpecName "kube-api-access-btrgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:41.456309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.456264    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10857aa4-9e02-4728-8581-c32e865769a6-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:41.459335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.459298    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btrgp\" (UniqueName: \"kubernetes.io/projected/10857aa4-9e02-4728-8581-c32e865769a6-kube-api-access-btrgp\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:41.643441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.643364    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"]
Oct 30 01:23:41.644065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644050    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ba1e782b-1121-4da0-a74d-18b8f624136a" containerName="revoke-keys"
Oct 30 01:23:41.644109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644066    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1e782b-1121-4da0-a74d-18b8f624136a" containerName="revoke-keys"
Oct 30 01:23:41.644109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644105    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b05c2802-6d7e-4e37-b594-87476c9a7083" containerName="revoke-keys"
Oct 30 01:23:41.644176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644110    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05c2802-6d7e-4e37-b594-87476c9a7083" containerName="revoke-keys"
Oct 30 01:23:41.644176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644123    4576 cpu_manager.go:414] "containerMap: removing container" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main"
Oct 30 01:23:41.644176 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644128    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main"
Oct 30 01:23:41.644332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644220    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05c2802-6d7e-4e37-b594-87476c9a7083" containerName="revoke-keys"
Oct 30 01:23:41.644332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644244    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="10857aa4-9e02-4728-8581-c32e865769a6" containerName="main"
Oct 30 01:23:41.644332 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.644254    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1e782b-1121-4da0-a74d-18b8f624136a" containerName="revoke-keys"
Oct 30 01:23:41.648087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.648071    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.668410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.668377    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"]
Oct 30 01:23:41.673278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.672208    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:41.673436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.673350    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:23:41.764921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.764881    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.765067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.765008    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.765116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.765099    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm57j\" (UniqueName: \"kubernetes.io/projected/699f02fd-8493-4143-983c-6b8767fcd27a-kube-api-access-fm57j\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.767914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.767883    4576 generic.go:356] "Generic (PLEG): container finished" podID="10857aa4-9e02-4728-8581-c32e865769a6" containerID="6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886" exitCode=0
Oct 30 01:23:41.768079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.767924    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" event={"ID":"10857aa4-9e02-4728-8581-c32e865769a6","Type":"ContainerDied","Data":"6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886"}
Oct 30 01:23:41.768079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.767943    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"
Oct 30 01:23:41.768079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.767975    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr" event={"ID":"10857aa4-9e02-4728-8581-c32e865769a6","Type":"ContainerDied","Data":"674b3957fd705791691ee34643cf0e48c72a21d2dfa8386b096f4dda2e5f6f6e"}
Oct 30 01:23:41.768079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.767998    4576 scope.go:122] "RemoveContainer" containerID="6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886"
Oct 30 01:23:41.782062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.782044    4576 scope.go:122] "RemoveContainer" containerID="6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886"
Oct 30 01:23:41.782390 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:41.782369    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886\": container with ID starting with 6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886 not found: ID does not exist" containerID="6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886"
Oct 30 01:23:41.782498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.782411    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886"} err="failed to get container status \"6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886\": rpc error: code = NotFound desc = could not find container \"6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886\": container with ID starting with 6820415e57fde1d95f6412dd6a7e5bd29cbe2a2ad9d7c7b94599cd964a449886 not found: ID does not exist"
Oct 30 01:23:41.827221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.827189    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"]
Oct 30 01:23:41.831865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.831839    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-cfe8f0/rate-test-model-cfe8f0-kserve-5ff748b677-f75fr"]
Oct 30 01:23:41.866361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.866329    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm57j\" (UniqueName: \"kubernetes.io/projected/699f02fd-8493-4143-983c-6b8767fcd27a-kube-api-access-fm57j\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.866576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.866559    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.866688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.866666    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.867260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.867219    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.869463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.869435    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.887263 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.887216    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm57j\" (UniqueName: \"kubernetes.io/projected/699f02fd-8493-4143-983c-6b8767fcd27a-kube-api-access-fm57j\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:41.967174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:41.967140    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:42.009599 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.009569    4576 scope.go:122] "RemoveContainer" containerID="df872acd4fd8bd2db4e4a925016a666a6854ac8c058eba4fe89884e044ed9d07"
Oct 30 01:23:42.023717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.023694    4576 scope.go:122] "RemoveContainer" containerID="a2d3ceeed105c9df35f4376f505be281fce2d5405ccb5f1b9ca1cb6185d29a8e"
Oct 30 01:23:42.031942 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.030596    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"]
Oct 30 01:23:42.040692 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.037991    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adf671fc-8zgws"]
Oct 30 01:23:42.111282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.111252    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10857aa4-9e02-4728-8581-c32e865769a6" path="/var/lib/kubelet/pods/10857aa4-9e02-4728-8581-c32e865769a6/volumes"
Oct 30 01:23:42.111815 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.111799    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc377c2-639b-4d6a-b41f-bd51682b6913" path="/var/lib/kubelet/pods/7bc377c2-639b-4d6a-b41f-bd51682b6913/volumes"
Oct 30 01:23:42.252601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.252514    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:42.278399 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:42.278368    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699f02fd_8493_4143_983c_6b8767fcd27a.slice/crio-13686a6e452aa7ef95523a8c31a6796b0b13fb9c3cf8099f5a9b3be54d5c07b4 WatchSource:0}: Error finding container 13686a6e452aa7ef95523a8c31a6796b0b13fb9c3cf8099f5a9b3be54d5c07b4: Status 404 returned error can't find the container with id 13686a6e452aa7ef95523a8c31a6796b0b13fb9c3cf8099f5a9b3be54d5c07b4
Oct 30 01:23:42.279002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.278929    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"]
Oct 30 01:23:42.372945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.372911    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:42.527381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.527295    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"]
Oct 30 01:23:42.528154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.528120    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy" containerID="cri-o://f3f3d5d6d847edf4235e651b796b4f0e72a8d17dc355ede9d708d3332c13dd1d" gracePeriod=30
Oct 30 01:23:42.538866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.538786    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"]
Oct 30 01:23:42.539336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.539300    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy" containerID="cri-o://0227d4a793de63e747ec4def1f82f9cf81e57f9685e92b4a4ffcae4d44c63958" gracePeriod=30
Oct 30 01:23:42.547220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.547195    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"]
Oct 30 01:23:42.548430 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.547610    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy" containerID="cri-o://a9764b3dc22e97996afce1237aef149798e7f66ca8d2153bc2306a2d3ca915f7" gracePeriod=30
Oct 30 01:23:42.554432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.554408    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"]
Oct 30 01:23:42.554656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.554630    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy" containerID="cri-o://f61e8926577dd07159887fe96849405224c5cd143bd38fd081aa8bd7802859be" gracePeriod=30
Oct 30 01:23:42.571878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.571851    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:42.772260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.760204    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:42.836258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:42.833854    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2" event={"ID":"699f02fd-8493-4143-983c-6b8767fcd27a","Type":"ContainerStarted","Data":"13686a6e452aa7ef95523a8c31a6796b0b13fb9c3cf8099f5a9b3be54d5c07b4"}
Oct 30 01:23:43.023928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.023893    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"]
Oct 30 01:23:43.028504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.028472    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.046556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.046472    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"]
Oct 30 01:23:43.121172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.121133    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.121409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.121366    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779kn\" (UniqueName: \"kubernetes.io/projected/b5a1ff87-40ed-4900-b069-0cc4258d31bd-kube-api-access-779kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.121658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.121635    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.223244 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.223202    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.223441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.223380    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779kn\" (UniqueName: \"kubernetes.io/projected/b5a1ff87-40ed-4900-b069-0cc4258d31bd-kube-api-access-779kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.223568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.223551    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.223862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.223843    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.226141 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.226110    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.243919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.243891    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779kn\" (UniqueName: \"kubernetes.io/projected/b5a1ff87-40ed-4900-b069-0cc4258d31bd-kube-api-access-779kn\") pod \"maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.345472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.345439    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:43.650167 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:43.650139    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a1ff87_40ed_4900_b069_0cc4258d31bd.slice/crio-5906c39ed63644328757c98cada243c23b21b8faa6bd959833d6efa00af09f34 WatchSource:0}: Error finding container 5906c39ed63644328757c98cada243c23b21b8faa6bd959833d6efa00af09f34: Status 404 returned error can't find the container with id 5906c39ed63644328757c98cada243c23b21b8faa6bd959833d6efa00af09f34
Oct 30 01:23:43.651000 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.650976    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"]
Oct 30 01:23:43.844485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.844445    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb" event={"ID":"b5a1ff87-40ed-4900-b069-0cc4258d31bd","Type":"ContainerStarted","Data":"5906c39ed63644328757c98cada243c23b21b8faa6bd959833d6efa00af09f34"}
Oct 30 01:23:43.846306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.846266    4576 generic.go:356] "Generic (PLEG): container finished" podID="699f02fd-8493-4143-983c-6b8767fcd27a" containerID="edb4c42663cbab195578374139295bf2ee31ac0259445c1916391be4a7efe4c4" exitCode=0
Oct 30 01:23:43.846447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:43.846307    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2" event={"ID":"699f02fd-8493-4143-983c-6b8767fcd27a","Type":"ContainerDied","Data":"edb4c42663cbab195578374139295bf2ee31ac0259445c1916391be4a7efe4c4"}
Oct 30 01:23:44.533468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:44.533434    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:44.649682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:44.649639    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:44.665652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:44.665609    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"]
Oct 30 01:23:44.784483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:44.784298    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:44.867159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:44.863942    4576 generic.go:356] "Generic (PLEG): container finished" podID="b5a1ff87-40ed-4900-b069-0cc4258d31bd" containerID="4207a2466fa7abe60f372a15c26309bd555c5dec256042c4d29efb455b4b7b88" exitCode=0
Oct 30 01:23:44.868038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:44.867997    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb" event={"ID":"b5a1ff87-40ed-4900-b069-0cc4258d31bd","Type":"ContainerDied","Data":"4207a2466fa7abe60f372a15c26309bd555c5dec256042c4d29efb455b4b7b88"}
Oct 30 01:23:44.896690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:44.896659    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:45.055688 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.055614    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:45.155843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.155810    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-xks-ca\") pod \"699f02fd-8493-4143-983c-6b8767fcd27a\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") "
Oct 30 01:23:45.156045 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.155851    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-openshift-ca\") pod \"699f02fd-8493-4143-983c-6b8767fcd27a\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") "
Oct 30 01:23:45.156045 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.156031    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699f02fd-8493-4143-983c-6b8767fcd27a-kube-api-access-fm57j\" (UniqueName: \"kubernetes.io/projected/699f02fd-8493-4143-983c-6b8767fcd27a-kube-api-access-fm57j\") pod \"699f02fd-8493-4143-983c-6b8767fcd27a\" (UID: \"699f02fd-8493-4143-983c-6b8767fcd27a\") "
Oct 30 01:23:45.156301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.156243    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-openshift-ca" pod "699f02fd-8493-4143-983c-6b8767fcd27a" (UID: "699f02fd-8493-4143-983c-6b8767fcd27a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:45.156627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.156599    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.158342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.158315    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-xks-ca" pod "699f02fd-8493-4143-983c-6b8767fcd27a" (UID: "699f02fd-8493-4143-983c-6b8767fcd27a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:45.158454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.158435    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699f02fd-8493-4143-983c-6b8767fcd27a-kube-api-access-fm57j" pod "699f02fd-8493-4143-983c-6b8767fcd27a" (UID: "699f02fd-8493-4143-983c-6b8767fcd27a"). InnerVolumeSpecName "kube-api-access-fm57j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:45.258477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.258441    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699f02fd-8493-4143-983c-6b8767fcd27a-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.258477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.258473    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm57j\" (UniqueName: \"kubernetes.io/projected/699f02fd-8493-4143-983c-6b8767fcd27a-kube-api-access-fm57j\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:45.876965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.876935    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"
Oct 30 01:23:45.877390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.876972    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2" event={"ID":"699f02fd-8493-4143-983c-6b8767fcd27a","Type":"ContainerDied","Data":"13686a6e452aa7ef95523a8c31a6796b0b13fb9c3cf8099f5a9b3be54d5c07b4"}
Oct 30 01:23:45.877390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:45.877020    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13686a6e452aa7ef95523a8c31a6796b0b13fb9c3cf8099f5a9b3be54d5c07b4"
Oct 30 01:23:46.066948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.066921    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:46.149960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.149882    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:46.173866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.173831    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a1ff87-40ed-4900-b069-0cc4258d31bd-kube-api-access-779kn\" (UniqueName: \"kubernetes.io/projected/b5a1ff87-40ed-4900-b069-0cc4258d31bd-kube-api-access-779kn\") pod \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") "
Oct 30 01:23:46.174040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.173902    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-xks-ca\") pod \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") "
Oct 30 01:23:46.174372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.174081    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-openshift-ca\") pod \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\" (UID: \"b5a1ff87-40ed-4900-b069-0cc4258d31bd\") "
Oct 30 01:23:46.175517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.174721    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-openshift-ca" pod "b5a1ff87-40ed-4900-b069-0cc4258d31bd" (UID: "b5a1ff87-40ed-4900-b069-0cc4258d31bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:46.175649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.175524    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.176817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.176791    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-xks-ca" pod "b5a1ff87-40ed-4900-b069-0cc4258d31bd" (UID: "b5a1ff87-40ed-4900-b069-0cc4258d31bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:46.176999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.176979    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a1ff87-40ed-4900-b069-0cc4258d31bd-kube-api-access-779kn" pod "b5a1ff87-40ed-4900-b069-0cc4258d31bd" (UID: "b5a1ff87-40ed-4900-b069-0cc4258d31bd"). InnerVolumeSpecName "kube-api-access-779kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:46.277163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.277119    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779kn\" (UniqueName: \"kubernetes.io/projected/b5a1ff87-40ed-4900-b069-0cc4258d31bd-kube-api-access-779kn\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.277163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.277155    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a1ff87-40ed-4900-b069-0cc4258d31bd-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.308999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.308919    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:46.515380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.515298    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:23:46.588848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.588822    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:23:46.589574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.589554    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-envoy\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.589637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.589625    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-data\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.589674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.589654    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istiod-ca-cert\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.589754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.589735    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-certs\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.589825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.589770    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-credential-socket\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.589897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.589882    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-token\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.589974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.589959    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-podinfo\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.590032 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.590007    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-kube-api-access-xdrx5\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-kube-api-access-xdrx5\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.590088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.589959    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-data" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.590088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.590050    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-certs" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.590210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.590115    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-socket\") pod \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\" (UID: \"ae8c9b6e-320d-4308-b9c2-01e9489a99fb\") "
Oct 30 01:23:46.590210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.590202    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istiod-ca-cert" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:46.590360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.590223    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-credential-socket" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.590519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.590493    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-socket" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.591434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.591319    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.591434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.591344    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.591434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.591360    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.591434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.591374    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.591434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.591389    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.593247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.593118    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-envoy" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.593707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.593679    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-kube-api-access-xdrx5" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "kube-api-access-xdrx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:46.594088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.594061    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-token" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:46.594560 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.594514    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-podinfo" pod "ae8c9b6e-320d-4308-b9c2-01e9489a99fb" (UID: "ae8c9b6e-320d-4308-b9c2-01e9489a99fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:46.626189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.626161    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.692952    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-podinfo\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693015    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-envoy\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693063    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-certs\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693136    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-kube-api-access-2ffw4\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-kube-api-access-2ffw4\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693169    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289432fe-271c-4f17-8ae3-edc5b8794d46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289432fe-271c-4f17-8ae3-edc5b8794d46-istiod-ca-cert\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693201    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istiod-ca-cert\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693261    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-socket\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693289    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-envoy\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693349    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-podinfo\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693409    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-credential-socket\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693519    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-certs\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693595    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-socket\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693641    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-token\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-token\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693671    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-data\") pod \"289432fe-271c-4f17-8ae3-edc5b8794d46\" (UID: \"289432fe-271c-4f17-8ae3-edc5b8794d46\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693818    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-token\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-token\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693870    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-credential-socket\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693904    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-data\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.693972    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-kube-api-access-wmv2v\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-kube-api-access-wmv2v\") pod \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\" (UID: \"f67ef6a4-517e-4b89-97f3-e8800f0cbefc\") "
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.694831    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.694854    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.694868    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrx5\" (UniqueName: \"kubernetes.io/projected/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-kube-api-access-xdrx5\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.694882    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c9b6e-320d-4308-b9c2-01e9489a99fb-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.695758    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-credential-socket" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.696410    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istiod-ca-cert" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.696714    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-certs" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.696750    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-certs" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.696960    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-socket" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.697575    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-data" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.698013    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-socket" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.698218    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289432fe-271c-4f17-8ae3-edc5b8794d46-istiod-ca-cert" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.698891    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-data" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.698922    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-envoy" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.698958    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-credential-socket" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.702667    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-podinfo" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:46.703843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.703676    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-kube-api-access-wmv2v" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "kube-api-access-wmv2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:46.711659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.711573    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-podinfo" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:46.711659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.711571    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-envoy" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:46.711659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.711594    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-token" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:46.711659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.711613    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-token" pod "f67ef6a4-517e-4b89-97f3-e8800f0cbefc" (UID: "f67ef6a4-517e-4b89-97f3-e8800f0cbefc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:46.713024 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.712995    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-kube-api-access-2ffw4" pod "289432fe-271c-4f17-8ae3-edc5b8794d46" (UID: "289432fe-271c-4f17-8ae3-edc5b8794d46"). InnerVolumeSpecName "kube-api-access-2ffw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:46.796916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796874    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.796916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796906    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.796916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796920    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796928    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796937    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796945    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796952    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796960    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796969    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796976    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796985    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmv2v\" (UniqueName: \"kubernetes.io/projected/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-kube-api-access-wmv2v\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.796995    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.797004    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289432fe-271c-4f17-8ae3-edc5b8794d46-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.797011    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.797019    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffw4\" (UniqueName: \"kubernetes.io/projected/289432fe-271c-4f17-8ae3-edc5b8794d46-kube-api-access-2ffw4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.797028    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289432fe-271c-4f17-8ae3-edc5b8794d46-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.797035    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.797257 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.797042    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67ef6a4-517e-4b89-97f3-e8800f0cbefc-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:46.893497 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.893447    4576 generic.go:356] "Generic (PLEG): container finished" podID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerID="e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219" exitCode=0
Oct 30 01:23:46.893966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.893697    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" event={"ID":"ae8c9b6e-320d-4308-b9c2-01e9489a99fb","Type":"ContainerDied","Data":"e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219"}
Oct 30 01:23:46.893966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.893792    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt" event={"ID":"ae8c9b6e-320d-4308-b9c2-01e9489a99fb","Type":"ContainerDied","Data":"018969fdcfe8ac6d57aa558df75df9ff062b7355839fc45375f88beacac0cae5"}
Oct 30 01:23:46.893966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.893837    4576 scope.go:122] "RemoveContainer" containerID="e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219"
Oct 30 01:23:46.894162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.894059    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"
Oct 30 01:23:46.896921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.896163    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb" event={"ID":"b5a1ff87-40ed-4900-b069-0cc4258d31bd","Type":"ContainerDied","Data":"5906c39ed63644328757c98cada243c23b21b8faa6bd959833d6efa00af09f34"}
Oct 30 01:23:46.896921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.896220    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5906c39ed63644328757c98cada243c23b21b8faa6bd959833d6efa00af09f34"
Oct 30 01:23:46.896921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.896320    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"
Oct 30 01:23:46.899280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.899251    4576 generic.go:356] "Generic (PLEG): container finished" podID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerID="cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585" exitCode=0
Oct 30 01:23:46.899423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.899333    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"
Oct 30 01:23:46.899423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.899342    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" event={"ID":"289432fe-271c-4f17-8ae3-edc5b8794d46","Type":"ContainerDied","Data":"cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585"}
Oct 30 01:23:46.899423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.899416    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc" event={"ID":"289432fe-271c-4f17-8ae3-edc5b8794d46","Type":"ContainerDied","Data":"faeb82641062c1c1b3a5e6541a8a6d8717a36053434e0aa9f1ea0d78fb2c98a1"}
Oct 30 01:23:46.901599 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.901574    4576 generic.go:356] "Generic (PLEG): container finished" podID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerID="56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac" exitCode=0
Oct 30 01:23:46.901927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.901899    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" event={"ID":"f67ef6a4-517e-4b89-97f3-e8800f0cbefc","Type":"ContainerDied","Data":"56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac"}
Oct 30 01:23:46.901927 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.901702    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"
Oct 30 01:23:46.902113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.901954    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf" event={"ID":"f67ef6a4-517e-4b89-97f3-e8800f0cbefc","Type":"ContainerDied","Data":"9ee076467e159418b06e18696adec4824875fd086c4a0d4ffcb112099bf93b68"}
Oct 30 01:23:46.909986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.909965    4576 scope.go:122] "RemoveContainer" containerID="e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219"
Oct 30 01:23:46.910366 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:46.910344    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219\": container with ID starting with e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219 not found: ID does not exist" containerID="e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219"
Oct 30 01:23:46.910458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.910378    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219"} err="failed to get container status \"e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219\": rpc error: code = NotFound desc = could not find container \"e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219\": container with ID starting with e23e1a6c2a1ca5dafea3ef62cb11aa899f7d99952ba9384efc0e2ec43962a219 not found: ID does not exist"
Oct 30 01:23:46.910458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.910403    4576 scope.go:122] "RemoveContainer" containerID="cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585"
Oct 30 01:23:46.923322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.923300    4576 scope.go:122] "RemoveContainer" containerID="c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f"
Oct 30 01:23:46.934852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.934830    4576 scope.go:122] "RemoveContainer" containerID="cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585"
Oct 30 01:23:46.935224 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:46.935202    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585\": container with ID starting with cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585 not found: ID does not exist" containerID="cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585"
Oct 30 01:23:46.935383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.935360    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585"} err="failed to get container status \"cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585\": rpc error: code = NotFound desc = could not find container \"cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585\": container with ID starting with cb070729e7c67e4918cf3d75796d44808042debb4f28139826c59955ab07e585 not found: ID does not exist"
Oct 30 01:23:46.935430 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.935386    4576 scope.go:122] "RemoveContainer" containerID="c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f"
Oct 30 01:23:46.935677 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:46.935660    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f\": container with ID starting with c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f not found: ID does not exist" containerID="c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f"
Oct 30 01:23:46.935724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.935682    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f"} err="failed to get container status \"c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f\": rpc error: code = NotFound desc = could not find container \"c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f\": container with ID starting with c9e1bb6e2ba1bc351e928a91fbff4670e486363411cc3417414375b9478ee76f not found: ID does not exist"
Oct 30 01:23:46.935724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.935698    4576 scope.go:122] "RemoveContainer" containerID="56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac"
Oct 30 01:23:46.946445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.946424    4576 scope.go:122] "RemoveContainer" containerID="56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac"
Oct 30 01:23:46.946691 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:46.946670    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac\": container with ID starting with 56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac not found: ID does not exist" containerID="56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac"
Oct 30 01:23:46.946755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.946701    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac"} err="failed to get container status \"56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac\": rpc error: code = NotFound desc = could not find container \"56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac\": container with ID starting with 56b34aed9eeaf4d59b0facd50e904d0a723afb8b4142ab6c910d55f6755776ac not found: ID does not exist"
Oct 30 01:23:46.989486 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:46.989408    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"]
Oct 30 01:23:47.003248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.003207    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-npjbc"]
Oct 30 01:23:47.044275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.044244    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"]
Oct 30 01:23:47.051049 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.051024    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-ts5cf"]
Oct 30 01:23:47.083759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.083729    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"]
Oct 30 01:23:47.087833 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.087809    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-07a3a8-openshift-default-5d699d59b5-tqnnt"]
Oct 30 01:23:47.917860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.917823    4576 generic.go:356] "Generic (PLEG): container finished" podID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerID="0227d4a793de63e747ec4def1f82f9cf81e57f9685e92b4a4ffcae4d44c63958" exitCode=0
Oct 30 01:23:47.917860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.917895    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" event={"ID":"0bd73bed-f9a0-436c-aead-a7aa427c9087","Type":"ContainerDied","Data":"0227d4a793de63e747ec4def1f82f9cf81e57f9685e92b4a4ffcae4d44c63958"}
Oct 30 01:23:47.920627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.920597    4576 generic.go:356] "Generic (PLEG): container finished" podID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerID="f3f3d5d6d847edf4235e651b796b4f0e72a8d17dc355ede9d708d3332c13dd1d" exitCode=0
Oct 30 01:23:47.920816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.920736    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" event={"ID":"81e867f5-4e74-42b1-941b-8d5e8e6b56d5","Type":"ContainerDied","Data":"f3f3d5d6d847edf4235e651b796b4f0e72a8d17dc355ede9d708d3332c13dd1d"}
Oct 30 01:23:47.920816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.920786    4576 scope.go:122] "RemoveContainer" containerID="0d372a03d8c6df1584fb849e279e326bdad9739794af16b76d2478d28302d10e"
Oct 30 01:23:47.925612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.925589    4576 generic.go:356] "Generic (PLEG): container finished" podID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerID="f61e8926577dd07159887fe96849405224c5cd143bd38fd081aa8bd7802859be" exitCode=0
Oct 30 01:23:47.925750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.925661    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" event={"ID":"4b572d61-32a8-4ca9-aa31-0640535d2897","Type":"ContainerDied","Data":"f61e8926577dd07159887fe96849405224c5cd143bd38fd081aa8bd7802859be"}
Oct 30 01:23:47.927953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.927825    4576 generic.go:356] "Generic (PLEG): container finished" podID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerID="a9764b3dc22e97996afce1237aef149798e7f66ca8d2153bc2306a2d3ca915f7" exitCode=0
Oct 30 01:23:47.927953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.927891    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" event={"ID":"a9026ba9-d689-41e5-b5df-f0371f725e9d","Type":"ContainerDied","Data":"a9764b3dc22e97996afce1237aef149798e7f66ca8d2153bc2306a2d3ca915f7"}
Oct 30 01:23:47.947291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.945276    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:23:47.959048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.959022    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:23:47.961342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.961063    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:23:47.968731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:47.968706    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:23:48.013948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.013911    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-credential-socket\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.014151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.013970    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-kube-api-access-s62cv\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-kube-api-access-s62cv\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.014151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014017    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-envoy\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.014151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014052    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-credential-socket\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.014151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014083    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-podinfo\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.014151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014109    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-envoy\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.014151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014145    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-token\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-token\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.014558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014188    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b572d61-32a8-4ca9-aa31-0640535d2897-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b572d61-32a8-4ca9-aa31-0640535d2897-istiod-ca-cert\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.014558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014305    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-socket\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.014558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014340    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9026ba9-d689-41e5-b5df-f0371f725e9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9026ba9-d689-41e5-b5df-f0371f725e9d-istiod-ca-cert\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.014558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014374    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-data\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.014558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014472    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-certs\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.014558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014519    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-token\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-token\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.014558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014555    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-certs\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.014943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014588    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-podinfo\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.014943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014669    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-data\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.014943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014731    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istiod-ca-cert\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.014943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014781    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-socket\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.014943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014812    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd73bed-f9a0-436c-aead-a7aa427c9087-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd73bed-f9a0-436c-aead-a7aa427c9087-istiod-ca-cert\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.014943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014841    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-credential-socket\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.014943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014870    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-data\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.014943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014915    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-kube-api-access-rwd8r\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-kube-api-access-rwd8r\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014959    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-envoy\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.014988    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-token\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-token\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015085    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-socket\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015121    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-data\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015149    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-podinfo\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015192    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-token\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-token\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015218    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-envoy\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015312    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-credential-socket\") pod \"0bd73bed-f9a0-436c-aead-a7aa427c9087\" (UID: \"0bd73bed-f9a0-436c-aead-a7aa427c9087\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015390    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-socket\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.015424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015419    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-certs\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.016046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015467    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-podinfo\") pod \"4b572d61-32a8-4ca9-aa31-0640535d2897\" (UID: \"4b572d61-32a8-4ca9-aa31-0640535d2897\") "
Oct 30 01:23:48.016046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015495    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-kube-api-access-wkv65\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-kube-api-access-wkv65\") pod \"a9026ba9-d689-41e5-b5df-f0371f725e9d\" (UID: \"a9026ba9-d689-41e5-b5df-f0371f725e9d\") "
Oct 30 01:23:48.016046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015591    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-certs\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.016046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.015637    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-kube-api-access-v4hqk\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-kube-api-access-v4hqk\") pod \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\" (UID: \"81e867f5-4e74-42b1-941b-8d5e8e6b56d5\") "
Oct 30 01:23:48.017213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.016666    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-socket" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.017213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.016903    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-data" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.017213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.017133    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-certs" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.017505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.017216    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-credential-socket" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.017505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.017407    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istiod-ca-cert" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:48.019601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.018345    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-credential-socket" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.019992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.019970    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-socket" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.020858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.020831    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-socket" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.021256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.021194    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9026ba9-d689-41e5-b5df-f0371f725e9d-istiod-ca-cert" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:48.021811 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.021630    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-data" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.022006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.021958    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-certs" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.022740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.022438    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-credential-socket" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.024085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.023505    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-certs" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.024085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.023544    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-envoy" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.024085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.023608    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-podinfo" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:48.024085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.023853    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b572d61-32a8-4ca9-aa31-0640535d2897-istiod-ca-cert" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:48.024484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.024453    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-socket" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.024742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.024714    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-data" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.024872 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.024850    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-token" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:48.024992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.024936    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-credential-socket" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.025067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.025000    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-podinfo" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:48.025196 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.025178    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-data" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.025379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.025361    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-certs" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.025472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.025403    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd73bed-f9a0-436c-aead-a7aa427c9087-istiod-ca-cert" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:48.026407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.026371    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-kube-api-access-wkv65" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "kube-api-access-wkv65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:48.027136 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.027112    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-envoy" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.027555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.027533    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-envoy" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.027661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.027575    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-podinfo" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:48.027661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.027631    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-podinfo" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:48.027796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.027671    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-token" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:48.027896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.027871    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-token" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:48.028207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.028184    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-kube-api-access-v4hqk" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "kube-api-access-v4hqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:48.028359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.028337    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-kube-api-access-s62cv" pod "4b572d61-32a8-4ca9-aa31-0640535d2897" (UID: "4b572d61-32a8-4ca9-aa31-0640535d2897"). InnerVolumeSpecName "kube-api-access-s62cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:48.028596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.028574    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-token" pod "a9026ba9-d689-41e5-b5df-f0371f725e9d" (UID: "a9026ba9-d689-41e5-b5df-f0371f725e9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:48.028933 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.028911    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-envoy" pod "81e867f5-4e74-42b1-941b-8d5e8e6b56d5" (UID: "81e867f5-4e74-42b1-941b-8d5e8e6b56d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:48.029104 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.029085    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-kube-api-access-rwd8r" pod "0bd73bed-f9a0-436c-aead-a7aa427c9087" (UID: "0bd73bed-f9a0-436c-aead-a7aa427c9087"). InnerVolumeSpecName "kube-api-access-rwd8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:48.115494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.115398    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" path="/var/lib/kubelet/pods/289432fe-271c-4f17-8ae3-edc5b8794d46/volumes"
Oct 30 01:23:48.116547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.116516    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" path="/var/lib/kubelet/pods/ae8c9b6e-320d-4308-b9c2-01e9489a99fb/volumes"
Oct 30 01:23:48.117271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117211    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" path="/var/lib/kubelet/pods/f67ef6a4-517e-4b89-97f3-e8800f0cbefc/volumes"
Oct 30 01:23:48.117889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117862    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117894    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117908    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117920    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117932    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkv65\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-kube-api-access-wkv65\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117947    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117960    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4hqk\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-kube-api-access-v4hqk\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117973    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117985    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62cv\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-kube-api-access-s62cv\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.117997    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118011    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118023    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118036    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118049    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118061    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b572d61-32a8-4ca9-aa31-0640535d2897-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118073    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118086    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9026ba9-d689-41e5-b5df-f0371f725e9d-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118100    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118113    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118125    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118319    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118347    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118360    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118372    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118387    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118400    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd73bed-f9a0-436c-aead-a7aa427c9087-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.118462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118413    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118564    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118595    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwd8r\" (UniqueName: \"kubernetes.io/projected/0bd73bed-f9a0-436c-aead-a7aa427c9087-kube-api-access-rwd8r\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118612    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118626    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9026ba9-d689-41e5-b5df-f0371f725e9d-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118640    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd73bed-f9a0-436c-aead-a7aa427c9087-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118773    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b572d61-32a8-4ca9-aa31-0640535d2897-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118788    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd73bed-f9a0-436c-aead-a7aa427c9087-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118801    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.119435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.118813    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e867f5-4e74-42b1-941b-8d5e8e6b56d5-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:48.189016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.188982    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:48.382564 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:48.382484    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:23:48.937131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.937096    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn" event={"ID":"4b572d61-32a8-4ca9-aa31-0640535d2897","Type":"ContainerDied","Data":"532b74196aa3a5eea6ff6d4f6658f60e8a67f284899b064c53eea4dc7b1f5cae"}
Oct 30 01:23:48.937657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.937104    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"
Oct 30 01:23:48.937657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.937165    4576 scope.go:122] "RemoveContainer" containerID="f61e8926577dd07159887fe96849405224c5cd143bd38fd081aa8bd7802859be"
Oct 30 01:23:48.939043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.939020    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"
Oct 30 01:23:48.939151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.939058    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq" event={"ID":"a9026ba9-d689-41e5-b5df-f0371f725e9d","Type":"ContainerDied","Data":"82cfeaf1db8c90b63576929ff0a60c40957ca4d058736776b7c20dbb0189d89d"}
Oct 30 01:23:48.941189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.941172    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"
Oct 30 01:23:48.941317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.941197    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t" event={"ID":"0bd73bed-f9a0-436c-aead-a7aa427c9087","Type":"ContainerDied","Data":"b2b4be964736d85746c1af0ef2a5c2e0ea65b34d7ec5700ec8402cc5b77ddc6b"}
Oct 30 01:23:48.942887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.942858    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"
Oct 30 01:23:48.942998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.942892    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46" event={"ID":"81e867f5-4e74-42b1-941b-8d5e8e6b56d5","Type":"ContainerDied","Data":"ae6db0d692ca24383fc12c3503f88cafaaab2b17469c52e06f70d63c6a76e833"}
Oct 30 01:23:48.951005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.950985    4576 scope.go:122] "RemoveContainer" containerID="a9764b3dc22e97996afce1237aef149798e7f66ca8d2153bc2306a2d3ca915f7"
Oct 30 01:23:48.961305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.961285    4576 scope.go:122] "RemoveContainer" containerID="0227d4a793de63e747ec4def1f82f9cf81e57f9685e92b4a4ffcae4d44c63958"
Oct 30 01:23:48.971762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:48.971740    4576 scope.go:122] "RemoveContainer" containerID="f3f3d5d6d847edf4235e651b796b4f0e72a8d17dc355ede9d708d3332c13dd1d"
Oct 30 01:23:49.012246 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.012202    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"]
Oct 30 01:23:49.026327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.026300    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-fll8t"]
Oct 30 01:23:49.053294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.053215    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"]
Oct 30 01:23:49.057178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.057153    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-f6smn"]
Oct 30 01:23:49.091269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.091223    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"]
Oct 30 01:23:49.095648 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.095618    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-h87rq"]
Oct 30 01:23:49.133721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.133688    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"]
Oct 30 01:23:49.137415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.137377    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c515e-openshift-default-7c9b898887-5kk46"]
Oct 30 01:23:49.679604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:49.679568    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:50.109629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:50.109595    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" path="/var/lib/kubelet/pods/0bd73bed-f9a0-436c-aead-a7aa427c9087/volumes"
Oct 30 01:23:50.110069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:50.110054    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" path="/var/lib/kubelet/pods/4b572d61-32a8-4ca9-aa31-0640535d2897/volumes"
Oct 30 01:23:50.110563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:50.110548    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" path="/var/lib/kubelet/pods/81e867f5-4e74-42b1-941b-8d5e8e6b56d5/volumes"
Oct 30 01:23:50.111357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:50.111342    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" path="/var/lib/kubelet/pods/a9026ba9-d689-41e5-b5df-f0371f725e9d/volumes"
Oct 30 01:23:50.876696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:50.876659    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"]
Oct 30 01:23:50.877157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:50.877124    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz" podUID="c592358a-3ff2-478f-a681-2a7c50d79afa" containerName="maas-api" containerID="cri-o://ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552" gracePeriod=30
Oct 30 01:23:50.956844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:50.956812    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:51.129351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.121595    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:51.232857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.232683    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:51.288125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.288100    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:23:51.376786 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.376740    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-metrics-certs\") pod \"c592358a-3ff2-478f-a681-2a7c50d79afa\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") "
Oct 30 01:23:51.377270 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.377225    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls\") pod \"c592358a-3ff2-478f-a681-2a7c50d79afa\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") "
Oct 30 01:23:51.377526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.377503    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c592358a-3ff2-478f-a681-2a7c50d79afa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c592358a-3ff2-478f-a681-2a7c50d79afa-openshift-service-ca\") pod \"c592358a-3ff2-478f-a681-2a7c50d79afa\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") "
Oct 30 01:23:51.377834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.377812    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c592358a-3ff2-478f-a681-2a7c50d79afa-kube-api-access-gsnz2\" (UniqueName: \"kubernetes.io/projected/c592358a-3ff2-478f-a681-2a7c50d79afa-kube-api-access-gsnz2\") pod \"c592358a-3ff2-478f-a681-2a7c50d79afa\" (UID: \"c592358a-3ff2-478f-a681-2a7c50d79afa\") "
Oct 30 01:23:51.378756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.377937    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c592358a-3ff2-478f-a681-2a7c50d79afa-openshift-service-ca" pod "c592358a-3ff2-478f-a681-2a7c50d79afa" (UID: "c592358a-3ff2-478f-a681-2a7c50d79afa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:51.408399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.408279    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-metrics-certs" pod "c592358a-3ff2-478f-a681-2a7c50d79afa" (UID: "c592358a-3ff2-478f-a681-2a7c50d79afa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:51.408954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.408896    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c592358a-3ff2-478f-a681-2a7c50d79afa-kube-api-access-gsnz2" pod "c592358a-3ff2-478f-a681-2a7c50d79afa" (UID: "c592358a-3ff2-478f-a681-2a7c50d79afa"). InnerVolumeSpecName "kube-api-access-gsnz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:51.416000 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.415954    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls" pod "c592358a-3ff2-478f-a681-2a7c50d79afa" (UID: "c592358a-3ff2-478f-a681-2a7c50d79afa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:51.487758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.487715    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:51.488372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.488344    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-maas-api-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:51.488669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.488379    4576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c592358a-3ff2-478f-a681-2a7c50d79afa-openshift-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:51.488669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.488393    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnz2\" (UniqueName: \"kubernetes.io/projected/c592358a-3ff2-478f-a681-2a7c50d79afa-kube-api-access-gsnz2\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:51.488669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.488406    4576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c592358a-3ff2-478f-a681-2a7c50d79afa-metrics-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:23:51.679760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.679650    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:51.707215 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.707176    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"]
Oct 30 01:23:51.711359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.711331    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c5a66-tqldx"]
Oct 30 01:23:51.942697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.942663    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:51.983411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.983378    4576 generic.go:356] "Generic (PLEG): container finished" podID="c592358a-3ff2-478f-a681-2a7c50d79afa" containerID="ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552" exitCode=0
Oct 30 01:23:51.983577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.983427    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"
Oct 30 01:23:51.983577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.983437    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz" event={"ID":"c592358a-3ff2-478f-a681-2a7c50d79afa","Type":"ContainerDied","Data":"ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552"}
Oct 30 01:23:51.983577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.983485    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz" event={"ID":"c592358a-3ff2-478f-a681-2a7c50d79afa","Type":"ContainerDied","Data":"262efdff133a1b2638f59a0b0965edf80d56e3c009571cedbda66235d5f8cca5"}
Oct 30 01:23:51.983577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.983514    4576 scope.go:122] "RemoveContainer" containerID="ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552"
Oct 30 01:23:51.995086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.995069    4576 scope.go:122] "RemoveContainer" containerID="ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552"
Oct 30 01:23:51.995427 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:23:51.995406    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552\": container with ID starting with ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552 not found: ID does not exist" containerID="ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552"
Oct 30 01:23:51.995510 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:51.995434    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552"} err="failed to get container status \"ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552\": rpc error: code = NotFound desc = could not find container \"ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552\": container with ID starting with ddc8881d37957664818aedc5c1fc5f79c8fd5c818ca9b70b51685db6db55e552 not found: ID does not exist"
Oct 30 01:23:52.028337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.028302    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"]
Oct 30 01:23:52.032260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.032215    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e4e5207-r9j4m"]
Oct 30 01:23:52.047942 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.047913    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"]
Oct 30 01:23:52.051528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.051505    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c5a66-7865c988dd-csnpz"]
Oct 30 01:23:52.109591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.109558    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a40906a-fc26-4db0-9373-18a08bd8454e" path="/var/lib/kubelet/pods/1a40906a-fc26-4db0-9373-18a08bd8454e/volumes"
Oct 30 01:23:52.110071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.110053    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1e782b-1121-4da0-a74d-18b8f624136a" path="/var/lib/kubelet/pods/ba1e782b-1121-4da0-a74d-18b8f624136a/volumes"
Oct 30 01:23:52.110677 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.110653    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c592358a-3ff2-478f-a681-2a7c50d79afa" path="/var/lib/kubelet/pods/c592358a-3ff2-478f-a681-2a7c50d79afa/volumes"
Oct 30 01:23:52.184434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.184396    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:52.254785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.254679    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"]
Oct 30 01:23:52.300603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.300567    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:52.927097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:52.927060    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:53.375729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:53.375696    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:53.492213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:53.492173    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:54.216646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:54.216611    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:55.791990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:55.791955    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:56.307120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:56.307084    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:56.868659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:56.868627    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:56.935924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:56.935893    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"]
Oct 30 01:23:56.949983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:56.949953    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-96bd13-2l2hb"]
Oct 30 01:23:57.026242 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:57.026191    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:57.392308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:57.392273    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:58.059708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.059666    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:58.114107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.114069    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a1ff87-40ed-4900-b069-0cc4258d31bd" path="/var/lib/kubelet/pods/b5a1ff87-40ed-4900-b069-0cc4258d31bd/volumes"
Oct 30 01:23:58.179112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.179078    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:58.316159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.316076    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:58.443611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.443575    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:58.779931 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.779889    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:58.847311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847276    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"]
Oct 30 01:23:58.847906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847889    4576 cpu_manager.go:414] "containerMap: removing container" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847907    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847927    4576 cpu_manager.go:414] "containerMap: removing container" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847932    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847942    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847948    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847958    4576 cpu_manager.go:414] "containerMap: removing container" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847963    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847978    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy"
Oct 30 01:23:58.847981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847983    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847990    4576 cpu_manager.go:414] "containerMap: removing container" podUID="c592358a-3ff2-478f-a681-2a7c50d79afa" containerName="maas-api"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.847996    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c592358a-3ff2-478f-a681-2a7c50d79afa" containerName="maas-api"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848012    4576 cpu_manager.go:414] "containerMap: removing container" podUID="699f02fd-8493-4143-983c-6b8767fcd27a" containerName="revoke-keys"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848018    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699f02fd-8493-4143-983c-6b8767fcd27a" containerName="revoke-keys"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848025    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b5a1ff87-40ed-4900-b069-0cc4258d31bd" containerName="revoke-keys"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848029    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a1ff87-40ed-4900-b069-0cc4258d31bd" containerName="revoke-keys"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848040    4576 cpu_manager.go:414] "containerMap: removing container" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848044    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848050    4576 cpu_manager.go:414] "containerMap: removing container" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848056    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848064    4576 cpu_manager.go:414] "containerMap: removing container" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848068    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848163    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="699f02fd-8493-4143-983c-6b8767fcd27a" containerName="revoke-keys"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848169    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a1ff87-40ed-4900-b069-0cc4258d31bd" containerName="revoke-keys"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848174    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848179    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848185    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="289432fe-271c-4f17-8ae3-edc5b8794d46" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848190    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9026ba9-d689-41e5-b5df-f0371f725e9d" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848196    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b572d61-32a8-4ca9-aa31-0640535d2897" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848202    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848208    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8c9b6e-320d-4308-b9c2-01e9489a99fb" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848214    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd73bed-f9a0-436c-aead-a7aa427c9087" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848220    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67ef6a4-517e-4b89-97f3-e8800f0cbefc" containerName="istio-proxy"
Oct 30 01:23:58.848525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.848226    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="c592358a-3ff2-478f-a681-2a7c50d79afa" containerName="maas-api"
Oct 30 01:23:58.853665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.853643    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:58.872714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.872688    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"]
Oct 30 01:23:58.882169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.882142    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:58.882383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.882318    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:23:58.957266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.940724    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzvm\" (UniqueName: \"kubernetes.io/projected/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-kube-api-access-gtzvm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:58.957266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.945829    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:58.957266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:58.946215    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:59.048311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.048180    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:59.048808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.048774    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzvm\" (UniqueName: \"kubernetes.io/projected/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-kube-api-access-gtzvm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:59.048907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.048843    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:59.049655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.049632    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:59.051817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.051792    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:59.069212 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.069183    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzvm\" (UniqueName: \"kubernetes.io/projected/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-kube-api-access-gtzvm\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:59.171983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.171951    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:23:59.485822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.485790    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:23:59.515696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.515662    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:59.521016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.520988    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"]
Oct 30 01:23:59.522093 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:23:59.522043    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8a880a_e9e2_4b92_9497_4fb7492a1bb2.slice/crio-896589120f26523c328f13a30084bb3c2529661219f26436deae6506c08a5582 WatchSource:0}: Error finding container 896589120f26523c328f13a30084bb3c2529661219f26436deae6506c08a5582: Status 404 returned error can't find the container with id 896589120f26523c328f13a30084bb3c2529661219f26436deae6506c08a5582
Oct 30 01:23:59.637665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.637634    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:23:59.686665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.686630    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"]
Oct 30 01:23:59.690431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.690404    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-24d5ce-s8zpf"]
Oct 30 01:23:59.711825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.711779    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm" podUID="8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" containerName="main" containerID="cri-o://299c4209eaebd6113a7d20be097c40ab900d789889da0b7bcc2ef309e7952d37" gracePeriod=45
Oct 30 01:23:59.935484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:23:59.935450    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:00.084821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.084782    4576 generic.go:356] "Generic (PLEG): container finished" podID="8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" containerID="299c4209eaebd6113a7d20be097c40ab900d789889da0b7bcc2ef309e7952d37" exitCode=0
Oct 30 01:24:00.085330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.084825    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm" event={"ID":"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b","Type":"ContainerDied","Data":"299c4209eaebd6113a7d20be097c40ab900d789889da0b7bcc2ef309e7952d37"}
Oct 30 01:24:00.085330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.084866    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm" event={"ID":"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b","Type":"ContainerDied","Data":"0fd350a2b496ed7c00e7153707afe19c58f543e0e5314dbe328e3d01578c80a9"}
Oct 30 01:24:00.085330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.084877    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd350a2b496ed7c00e7153707afe19c58f543e0e5314dbe328e3d01578c80a9"
Oct 30 01:24:00.086955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.086926    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f" event={"ID":"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2","Type":"ContainerStarted","Data":"fd11c949e4d49a1a6b1ec18d16c768b3dc77e2da61221c0cfe32df8b57d70e14"}
Oct 30 01:24:00.087105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.086986    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f" event={"ID":"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2","Type":"ContainerStarted","Data":"896589120f26523c328f13a30084bb3c2529661219f26436deae6506c08a5582"}
Oct 30 01:24:00.093110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.093089    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:24:00.109686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.109660    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d29976-e999-42de-865e-5b31caaa9032" path="/var/lib/kubelet/pods/e4d29976-e999-42de-865e-5b31caaa9032/volumes"
Oct 30 01:24:00.132998 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.132949    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f" podStartSLOduration=2.132935059 podStartE2EDuration="2.132935059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:00.130692903 +0000 UTC m=+2720.202016739" watchObservedRunningTime="2026-10-30 01:24:00.132935059 +0000 UTC m=+2720.204258894"
Oct 30 01:24:00.164030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.163997    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:00.177162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.175807    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tmp-dir\") pod \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") "
Oct 30 01:24:00.177162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.175923    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tls-certs\") pod \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") "
Oct 30 01:24:00.177162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.175980    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-model-cache\") pod \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") "
Oct 30 01:24:00.177162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.176058    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tmp-dir" pod "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" (UID: "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.177162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.176162    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-dshm\") pod \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") "
Oct 30 01:24:00.177162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.176362    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-home\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-home\") pod \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") "
Oct 30 01:24:00.177162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.176410    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-kube-api-access-nb2vw\" (UniqueName: \"kubernetes.io/projected/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-kube-api-access-nb2vw\") pod \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\" (UID: \"8a8c0f87-19cd-4bf9-8e32-cc907bfb118b\") "
Oct 30 01:24:00.177162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.176427    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-model-cache" pod "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" (UID: "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.177847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.177382    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.177847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.177414    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.184667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.180267    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tls-certs" pod "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" (UID: "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:00.184667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.181726    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-kube-api-access-nb2vw" pod "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" (UID: "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b"). InnerVolumeSpecName "kube-api-access-nb2vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:00.184667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.182131    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-dshm" pod "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" (UID: "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.185155 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.185131    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-home" pod "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" (UID: "8a8c0f87-19cd-4bf9-8e32-cc907bfb118b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:00.279180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.279140    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.279180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.279171    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2vw\" (UniqueName: \"kubernetes.io/projected/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-kube-api-access-nb2vw\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.279180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.279183    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.279180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.279192    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:00.673254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.672508    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:00.805314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:00.798156    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:01.130307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:01.129600    4576 generic.go:356] "Generic (PLEG): container finished" podID="4b8a880a-e9e2-4b92-9497-4fb7492a1bb2" containerID="fd11c949e4d49a1a6b1ec18d16c768b3dc77e2da61221c0cfe32df8b57d70e14" exitCode=0
Oct 30 01:24:01.130307 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:01.130009    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"
Oct 30 01:24:01.138306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:01.134418    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f" event={"ID":"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2","Type":"ContainerDied","Data":"fd11c949e4d49a1a6b1ec18d16c768b3dc77e2da61221c0cfe32df8b57d70e14"}
Oct 30 01:24:01.326259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:01.323858    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"]
Oct 30 01:24:01.353254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:01.349132    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-60f38f/auth-test-model-60f38f-kserve-54d4d77bfc-6t9sm"]
Oct 30 01:24:02.109067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.109032    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" path="/var/lib/kubelet/pods/8a8c0f87-19cd-4bf9-8e32-cc907bfb118b/volumes"
Oct 30 01:24:02.307580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.307553    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:24:02.310683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.310654    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:02.431611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.431526    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-openshift-ca\") pod \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") "
Oct 30 01:24:02.431611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.431603    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-xks-ca\") pod \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") "
Oct 30 01:24:02.431818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.431752    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-kube-api-access-gtzvm\" (UniqueName: \"kubernetes.io/projected/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-kube-api-access-gtzvm\") pod \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\" (UID: \"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2\") "
Oct 30 01:24:02.432019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.431994    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-openshift-ca" pod "4b8a880a-e9e2-4b92-9497-4fb7492a1bb2" (UID: "4b8a880a-e9e2-4b92-9497-4fb7492a1bb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:02.432633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.432607    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.434114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.434089    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-xks-ca" pod "4b8a880a-e9e2-4b92-9497-4fb7492a1bb2" (UID: "4b8a880a-e9e2-4b92-9497-4fb7492a1bb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:02.434202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.434109    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-kube-api-access-gtzvm" pod "4b8a880a-e9e2-4b92-9497-4fb7492a1bb2" (UID: "4b8a880a-e9e2-4b92-9497-4fb7492a1bb2"). InnerVolumeSpecName "kube-api-access-gtzvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:02.535528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.535343    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.535528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.535379    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzvm\" (UniqueName: \"kubernetes.io/projected/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2-kube-api-access-gtzvm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.879195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:02.879080    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:03.106098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.106066    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:03.146359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.146279    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"
Oct 30 01:24:03.146496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.146269    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f" event={"ID":"4b8a880a-e9e2-4b92-9497-4fb7492a1bb2","Type":"ContainerDied","Data":"896589120f26523c328f13a30084bb3c2529661219f26436deae6506c08a5582"}
Oct 30 01:24:03.146496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.146408    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="896589120f26523c328f13a30084bb3c2529661219f26436deae6506c08a5582"
Oct 30 01:24:03.155500 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.155474    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"]
Oct 30 01:24:03.169085 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.169060    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3c2596-wzhzl"]
Oct 30 01:24:03.219472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.219440    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:03.558361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.558320    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:03.720100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.720062    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:03.855461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:03.855375    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:04.109816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.109739    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46eff990-edaf-4db0-bf11-ecf53146ed0d" path="/var/lib/kubelet/pods/46eff990-edaf-4db0-bf11-ecf53146ed0d/volumes"
Oct 30 01:24:04.294831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.294796    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"]
Oct 30 01:24:04.295702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.295681    4576 cpu_manager.go:414] "containerMap: removing container" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy"
Oct 30 01:24:04.295756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.295704    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e867f5-4e74-42b1-941b-8d5e8e6b56d5" containerName="istio-proxy"
Oct 30 01:24:04.295756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.295721    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" containerName="main"
Oct 30 01:24:04.295756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.295726    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" containerName="main"
Oct 30 01:24:04.295756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.295735    4576 cpu_manager.go:414] "containerMap: removing container" podUID="4b8a880a-e9e2-4b92-9497-4fb7492a1bb2" containerName="revoke-keys"
Oct 30 01:24:04.295756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.295741    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8a880a-e9e2-4b92-9497-4fb7492a1bb2" containerName="revoke-keys"
Oct 30 01:24:04.295937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.295865    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8c0f87-19cd-4bf9-8e32-cc907bfb118b" containerName="main"
Oct 30 01:24:04.295937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.295873    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8a880a-e9e2-4b92-9497-4fb7492a1bb2" containerName="revoke-keys"
Oct 30 01:24:04.300590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.300570    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.315737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.315715    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"]
Oct 30 01:24:04.325160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.325001    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:04.325160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.325075    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:24:04.397004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.396920    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.397004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.396958    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpr89\" (UniqueName: \"kubernetes.io/projected/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-kube-api-access-mpr89\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.397179 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.397142    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.498573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.498535    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.498730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.498689    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.498730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.498707    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpr89\" (UniqueName: \"kubernetes.io/projected/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-kube-api-access-mpr89\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.499350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.499323    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.501153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.501125    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.520083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.520056    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpr89\" (UniqueName: \"kubernetes.io/projected/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-kube-api-access-mpr89\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.615843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.615814    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:04.926928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:04.926899    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"]
Oct 30 01:24:04.927753 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:04.927726    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be111cd_3a03_4ff0_bcf7_f5bc59ecb7e6.slice/crio-e0731f82eef4ff4d9b57dcb9d3d51faa0fcd2d407a12747872fd5b8c0f14f838 WatchSource:0}: Error finding container e0731f82eef4ff4d9b57dcb9d3d51faa0fcd2d407a12747872fd5b8c0f14f838: Status 404 returned error can't find the container with id e0731f82eef4ff4d9b57dcb9d3d51faa0fcd2d407a12747872fd5b8c0f14f838
Oct 30 01:24:05.168948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:05.168895    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2" event={"ID":"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6","Type":"ContainerStarted","Data":"e0731f82eef4ff4d9b57dcb9d3d51faa0fcd2d407a12747872fd5b8c0f14f838"}
Oct 30 01:24:05.203825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:05.203793    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:05.952522 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:05.952486    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:06.062183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.062146    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:06.191425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.191221    4576 generic.go:356] "Generic (PLEG): container finished" podID="3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6" containerID="79df3450ea745b6341c3d049528103edae6aff9e14c4391ab52e19bfbc5978f6" exitCode=0
Oct 30 01:24:06.191886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.191653    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2" event={"ID":"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6","Type":"ContainerDied","Data":"79df3450ea745b6341c3d049528103edae6aff9e14c4391ab52e19bfbc5978f6"}
Oct 30 01:24:06.224847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.224757    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"]
Oct 30 01:24:06.225610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.225575    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" containerID="cri-o://3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40" gracePeriod=30
Oct 30 01:24:06.272097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.272052    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:06.711983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.711936    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"]
Oct 30 01:24:06.716374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.716351    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.735636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.735604    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"]
Oct 30 01:24:06.836092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.836058    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.836294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.836122    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.836294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.836268    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2l6\" (UniqueName: \"kubernetes.io/projected/92ab0966-c954-43f1-8478-d31e88703b0e-kube-api-access-rb2l6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.937169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.937135    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.937406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.937210    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.937406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.937347    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2l6\" (UniqueName: \"kubernetes.io/projected/92ab0966-c954-43f1-8478-d31e88703b0e-kube-api-access-rb2l6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.938041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.937954    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.939729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.939708    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:06.958033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:06.958006    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2l6\" (UniqueName: \"kubernetes.io/projected/92ab0966-c954-43f1-8478-d31e88703b0e-kube-api-access-rb2l6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:07.033654 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.033558    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:07.283163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.282721    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv" podUID="fdc83f61-3437-423b-b0e6-096c8f017880" containerName="main" containerID="cri-o://113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5" gracePeriod=45
Oct 30 01:24:07.337823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.337792    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"]
Oct 30 01:24:07.344732 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:07.344690    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ab0966_c954_43f1_8478_d31e88703b0e.slice/crio-087f26d986dea0fc80818c390e3249c42c02e2b3834f8ed39d680488eb6833e4 WatchSource:0}: Error finding container 087f26d986dea0fc80818c390e3249c42c02e2b3834f8ed39d680488eb6833e4: Status 404 returned error can't find the container with id 087f26d986dea0fc80818c390e3249c42c02e2b3834f8ed39d680488eb6833e4
Oct 30 01:24:07.399349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.399323    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:07.475094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.475053    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:07.548034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.547948    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-kube-api-access-mpr89\" (UniqueName: \"kubernetes.io/projected/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-kube-api-access-mpr89\") pod \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") "
Oct 30 01:24:07.548207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.548070    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-openshift-ca\") pod \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") "
Oct 30 01:24:07.548314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.548275    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-xks-ca\") pod \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\" (UID: \"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6\") "
Oct 30 01:24:07.548581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.548546    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-openshift-ca" pod "3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6" (UID: "3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:07.549522 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.549493    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:07.568631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.568592    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-kube-api-access-mpr89" pod "3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6" (UID: "3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6"). InnerVolumeSpecName "kube-api-access-mpr89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:07.569335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.569306    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-xks-ca" pod "3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6" (UID: "3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:07.599415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.599393    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:24:07.650583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.650526    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc83f61-3437-423b-b0e6-096c8f017880-tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc83f61-3437-423b-b0e6-096c8f017880-tls-certs\") pod \"fdc83f61-3437-423b-b0e6-096c8f017880\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") "
Oct 30 01:24:07.650788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.650643    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-home\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-home\") pod \"fdc83f61-3437-423b-b0e6-096c8f017880\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") "
Oct 30 01:24:07.650788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.650695    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-dshm\") pod \"fdc83f61-3437-423b-b0e6-096c8f017880\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") "
Oct 30 01:24:07.650788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.650741    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-tmp-dir\") pod \"fdc83f61-3437-423b-b0e6-096c8f017880\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") "
Oct 30 01:24:07.650982 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.650860    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc83f61-3437-423b-b0e6-096c8f017880-kube-api-access-rsm9t\" (UniqueName: \"kubernetes.io/projected/fdc83f61-3437-423b-b0e6-096c8f017880-kube-api-access-rsm9t\") pod \"fdc83f61-3437-423b-b0e6-096c8f017880\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") "
Oct 30 01:24:07.651047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.651005    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-model-cache\") pod \"fdc83f61-3437-423b-b0e6-096c8f017880\" (UID: \"fdc83f61-3437-423b-b0e6-096c8f017880\") "
Oct 30 01:24:07.651757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.651729    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpr89\" (UniqueName: \"kubernetes.io/projected/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-kube-api-access-mpr89\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:07.651890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.651761    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:07.652208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.652166    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-tmp-dir" pod "fdc83f61-3437-423b-b0e6-096c8f017880" (UID: "fdc83f61-3437-423b-b0e6-096c8f017880"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:07.652431 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.652405    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-model-cache" pod "fdc83f61-3437-423b-b0e6-096c8f017880" (UID: "fdc83f61-3437-423b-b0e6-096c8f017880"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:07.660288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.653359    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-home" pod "fdc83f61-3437-423b-b0e6-096c8f017880" (UID: "fdc83f61-3437-423b-b0e6-096c8f017880"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:07.660288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.653727    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc83f61-3437-423b-b0e6-096c8f017880-tls-certs" pod "fdc83f61-3437-423b-b0e6-096c8f017880" (UID: "fdc83f61-3437-423b-b0e6-096c8f017880"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:07.660288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.654084    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-dshm" pod "fdc83f61-3437-423b-b0e6-096c8f017880" (UID: "fdc83f61-3437-423b-b0e6-096c8f017880"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:07.664412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.664384    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc83f61-3437-423b-b0e6-096c8f017880-kube-api-access-rsm9t" pod "fdc83f61-3437-423b-b0e6-096c8f017880" (UID: "fdc83f61-3437-423b-b0e6-096c8f017880"). InnerVolumeSpecName "kube-api-access-rsm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:07.759596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.759555    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc83f61-3437-423b-b0e6-096c8f017880-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:07.759596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.759588    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:07.759596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.759601    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:07.759936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.759612    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:07.759936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.759624    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsm9t\" (UniqueName: \"kubernetes.io/projected/fdc83f61-3437-423b-b0e6-096c8f017880-kube-api-access-rsm9t\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:07.759936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:07.759634    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc83f61-3437-423b-b0e6-096c8f017880-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:08.209382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.209349    4576 generic.go:356] "Generic (PLEG): container finished" podID="fdc83f61-3437-423b-b0e6-096c8f017880" containerID="113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5" exitCode=0
Oct 30 01:24:08.209854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.209408    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"
Oct 30 01:24:08.209854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.209430    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv" event={"ID":"fdc83f61-3437-423b-b0e6-096c8f017880","Type":"ContainerDied","Data":"113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5"}
Oct 30 01:24:08.209854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.209473    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv" event={"ID":"fdc83f61-3437-423b-b0e6-096c8f017880","Type":"ContainerDied","Data":"b20381b50de3e79512b7fa5e2201be0e0a8f467f25715c0079be43b0632bc7e7"}
Oct 30 01:24:08.209854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.209495    4576 scope.go:122] "RemoveContainer" containerID="113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5"
Oct 30 01:24:08.211203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.211185    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"
Oct 30 01:24:08.211391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.211275    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2" event={"ID":"3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6","Type":"ContainerDied","Data":"e0731f82eef4ff4d9b57dcb9d3d51faa0fcd2d407a12747872fd5b8c0f14f838"}
Oct 30 01:24:08.211391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.211311    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0731f82eef4ff4d9b57dcb9d3d51faa0fcd2d407a12747872fd5b8c0f14f838"
Oct 30 01:24:08.213923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.213896    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4" event={"ID":"92ab0966-c954-43f1-8478-d31e88703b0e","Type":"ContainerStarted","Data":"6878fec1aa7b37040782da01309aa041cc22159fe18041282d3c4ba014154208"}
Oct 30 01:24:08.214063 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.213960    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4" event={"ID":"92ab0966-c954-43f1-8478-d31e88703b0e","Type":"ContainerStarted","Data":"087f26d986dea0fc80818c390e3249c42c02e2b3834f8ed39d680488eb6833e4"}
Oct 30 01:24:08.222682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.222660    4576 scope.go:122] "RemoveContainer" containerID="113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5"
Oct 30 01:24:08.222966 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:08.222949    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5\": container with ID starting with 113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5 not found: ID does not exist" containerID="113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5"
Oct 30 01:24:08.223049 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.222972    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5"} err="failed to get container status \"113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5\": rpc error: code = NotFound desc = could not find container \"113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5\": container with ID starting with 113bfce96d6d82698feaa52de8ec7568d809b0c09446fa65fb3d7fe53f25cae5 not found: ID does not exist"
Oct 30 01:24:08.266999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.266967    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"]
Oct 30 01:24:08.280838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.280810    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-954bf6/auth-test-model-954bf6-kserve-795b685bd9-47wqv"]
Oct 30 01:24:08.320015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.319964    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4" podStartSLOduration=2.319948273 podStartE2EDuration="2.319948273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:08.317351027 +0000 UTC m=+2728.388674862" watchObservedRunningTime="2026-10-30 01:24:08.319948273 +0000 UTC m=+2728.391272108"
Oct 30 01:24:08.381652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.381621    4576 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:24:08.381824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:08.381676    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:24:09.160156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.160114    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:09.244435 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.244388    4576 generic.go:356] "Generic (PLEG): container finished" podID="92ab0966-c954-43f1-8478-d31e88703b0e" containerID="6878fec1aa7b37040782da01309aa041cc22159fe18041282d3c4ba014154208" exitCode=0
Oct 30 01:24:09.244910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.244474    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4" event={"ID":"92ab0966-c954-43f1-8478-d31e88703b0e","Type":"ContainerDied","Data":"6878fec1aa7b37040782da01309aa041cc22159fe18041282d3c4ba014154208"}
Oct 30 01:24:09.295566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.295526    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:09.313363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.313332    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"]
Oct 30 01:24:09.415222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.415085    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:09.554292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.554209    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:09.574290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.573217    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"]
Oct 30 01:24:09.574290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.573502    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy" containerID="cri-o://7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f" gracePeriod=30
Oct 30 01:24:09.669273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:09.669164    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:10.110443 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.110411    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc83f61-3437-423b-b0e6-096c8f017880" path="/var/lib/kubelet/pods/fdc83f61-3437-423b-b0e6-096c8f017880/volumes"
Oct 30 01:24:10.217890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.217607    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:10.334377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.334341    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:10.432442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.432417    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:10.507123 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.507090    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-openshift-ca\") pod \"92ab0966-c954-43f1-8478-d31e88703b0e\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") "
Oct 30 01:24:10.507299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.507247    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ab0966-c954-43f1-8478-d31e88703b0e-kube-api-access-rb2l6\" (UniqueName: \"kubernetes.io/projected/92ab0966-c954-43f1-8478-d31e88703b0e-kube-api-access-rb2l6\") pod \"92ab0966-c954-43f1-8478-d31e88703b0e\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") "
Oct 30 01:24:10.507457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.507405    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-xks-ca\") pod \"92ab0966-c954-43f1-8478-d31e88703b0e\" (UID: \"92ab0966-c954-43f1-8478-d31e88703b0e\") "
Oct 30 01:24:10.507580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.507552    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-openshift-ca" pod "92ab0966-c954-43f1-8478-d31e88703b0e" (UID: "92ab0966-c954-43f1-8478-d31e88703b0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:10.508028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.508001    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:10.509984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.509955    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ab0966-c954-43f1-8478-d31e88703b0e-kube-api-access-rb2l6" pod "92ab0966-c954-43f1-8478-d31e88703b0e" (UID: "92ab0966-c954-43f1-8478-d31e88703b0e"). InnerVolumeSpecName "kube-api-access-rb2l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:10.510271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.510202    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-xks-ca" pod "92ab0966-c954-43f1-8478-d31e88703b0e" (UID: "92ab0966-c954-43f1-8478-d31e88703b0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:10.609731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.609697    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb2l6\" (UniqueName: \"kubernetes.io/projected/92ab0966-c954-43f1-8478-d31e88703b0e-kube-api-access-rb2l6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:10.609731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:10.609727    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ab0966-c954-43f1-8478-d31e88703b0e-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.071555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.071518    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:11.261631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.261607    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"
Oct 30 01:24:11.261768 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.261649    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4" event={"ID":"92ab0966-c954-43f1-8478-d31e88703b0e","Type":"ContainerDied","Data":"087f26d986dea0fc80818c390e3249c42c02e2b3834f8ed39d680488eb6833e4"}
Oct 30 01:24:11.261768 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.261705    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="087f26d986dea0fc80818c390e3249c42c02e2b3834f8ed39d680488eb6833e4"
Oct 30 01:24:11.516930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.516908    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:24:11.624035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.623948    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-kube-api-access-c8df7\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-kube-api-access-c8df7\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624035 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.623986    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/900b3754-1cef-4071-b077-2663fcd55336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b3754-1cef-4071-b077-2663fcd55336-istio-podinfo\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624080    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-socket\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624105    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-credential-socket\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624123    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-envoy\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624147    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900b3754-1cef-4071-b077-2663fcd55336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b3754-1cef-4071-b077-2663fcd55336-istiod-ca-cert\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624182    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-certs\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624285    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-data\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624354    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-istio-token\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-istio-token\") pod \"900b3754-1cef-4071-b077-2663fcd55336\" (UID: \"900b3754-1cef-4071-b077-2663fcd55336\") "
Oct 30 01:24:11.624557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624407    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-socket" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.624557 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624511    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-credential-socket" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.624779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624635    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-data" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.624779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.624676    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-certs" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.625127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.625087    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900b3754-1cef-4071-b077-2663fcd55336-istiod-ca-cert" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:11.625463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.625253    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.625463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.625274    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.625463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.625288    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/900b3754-1cef-4071-b077-2663fcd55336-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.625463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.625301    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.625463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.625314    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.626705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.626680    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-envoy" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:11.626844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.626680    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/900b3754-1cef-4071-b077-2663fcd55336-istio-podinfo" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:11.626844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.626708    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-kube-api-access-c8df7" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "kube-api-access-c8df7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:11.626919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.626871    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-istio-token" pod "900b3754-1cef-4071-b077-2663fcd55336" (UID: "900b3754-1cef-4071-b077-2663fcd55336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:11.726154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.726120    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8df7\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-kube-api-access-c8df7\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.726154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.726149    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/900b3754-1cef-4071-b077-2663fcd55336-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.726154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.726158    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/900b3754-1cef-4071-b077-2663fcd55336-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:11.726438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:11.726166    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/900b3754-1cef-4071-b077-2663fcd55336-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:12.271607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.271573    4576 generic.go:356] "Generic (PLEG): container finished" podID="900b3754-1cef-4071-b077-2663fcd55336" containerID="3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40" exitCode=0
Oct 30 01:24:12.271792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.271620    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"
Oct 30 01:24:12.271792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.271661    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" event={"ID":"900b3754-1cef-4071-b077-2663fcd55336","Type":"ContainerDied","Data":"3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40"}
Oct 30 01:24:12.271792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.271725    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc" event={"ID":"900b3754-1cef-4071-b077-2663fcd55336","Type":"ContainerDied","Data":"d53df7cc0b372fd15c77f132ab0705f2a8e66211093c05b21b11aa2d3ef8c093"}
Oct 30 01:24:12.271792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.271754    4576 scope.go:122] "RemoveContainer" containerID="3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40"
Oct 30 01:24:12.285171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.285153    4576 scope.go:122] "RemoveContainer" containerID="3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40"
Oct 30 01:24:12.285493 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:12.285474    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40\": container with ID starting with 3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40 not found: ID does not exist" containerID="3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40"
Oct 30 01:24:12.285535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.285501    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40"} err="failed to get container status \"3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40\": rpc error: code = NotFound desc = could not find container \"3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40\": container with ID starting with 3f452ae0c48d1731e512c3d274d040bb3eff03156a8f4d4240f13a8810286d40 not found: ID does not exist"
Oct 30 01:24:12.344847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.344815    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"]
Oct 30 01:24:12.358458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.358426    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24d5ce-openshift-default-55fb54db69-k4fqc"]
Oct 30 01:24:12.557795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:12.557750    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:14.097953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:14.097923    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:14.108739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:14.108714    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900b3754-1cef-4071-b077-2663fcd55336" path="/var/lib/kubelet/pods/900b3754-1cef-4071-b077-2663fcd55336/volumes"
Oct 30 01:24:14.313672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:14.313614    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff" podUID="d50140ed-d37d-4110-8331-71bf08574258" containerName="payload-processing" containerID="cri-o://c93f6425dfad5b07022ec4f1a8178f7ca4998ce4f44bc13459260d866e76576a" gracePeriod=25
Oct 30 01:24:14.748720 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:14.748687    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"]
Oct 30 01:24:14.785704 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:14.785670    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"]
Oct 30 01:24:14.896488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:14.896454    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 01:24:14.901536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:14.901466    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:15.028384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.028307    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:15.077079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077045    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-kube-api-access-92xqr\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-kube-api-access-92xqr\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077097    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-socket\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077168    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-data\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077219    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0987db5-792d-4300-a535-2dd9b7a17cb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0987db5-792d-4300-a535-2dd9b7a17cb3-istiod-ca-cert\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077312    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-certs\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077419    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-podinfo\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077450    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-socket" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:15.077518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077470    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-credential-socket\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077568    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-token\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-token\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077587    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-data" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:15.077740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077661    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-certs" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:15.077740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077713    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0987db5-792d-4300-a535-2dd9b7a17cb3-istiod-ca-cert" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:15.077936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077602    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-envoy\") pod \"f0987db5-792d-4300-a535-2dd9b7a17cb3\" (UID: \"f0987db5-792d-4300-a535-2dd9b7a17cb3\") "
Oct 30 01:24:15.077936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.077774    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-credential-socket" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:15.078372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.078350    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.078464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.078373    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.078464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.078386    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.078464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.078399    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0987db5-792d-4300-a535-2dd9b7a17cb3-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.078464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.078410    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.080294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.080224    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-podinfo" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:15.080417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.080293    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-kube-api-access-92xqr" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "kube-api-access-92xqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:15.080476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.080407    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-envoy" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:15.080476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.080431    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-token" pod "f0987db5-792d-4300-a535-2dd9b7a17cb3" (UID: "f0987db5-792d-4300-a535-2dd9b7a17cb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:15.179696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.179617    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.179696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.179650    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.179696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.179665    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0987db5-792d-4300-a535-2dd9b7a17cb3-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.179696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.179677    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92xqr\" (UniqueName: \"kubernetes.io/projected/f0987db5-792d-4300-a535-2dd9b7a17cb3-kube-api-access-92xqr\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:15.304274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.304220    4576 generic.go:356] "Generic (PLEG): container finished" podID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerID="7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f" exitCode=0
Oct 30 01:24:15.304464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.304311    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" event={"ID":"f0987db5-792d-4300-a535-2dd9b7a17cb3","Type":"ContainerDied","Data":"7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f"}
Oct 30 01:24:15.304464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.304327    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"
Oct 30 01:24:15.304464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.304366    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft" event={"ID":"f0987db5-792d-4300-a535-2dd9b7a17cb3","Type":"ContainerDied","Data":"b5d07ce6b80863e41819f57e418f2908752bf137a57801facc616bff49a6f2fa"}
Oct 30 01:24:15.304464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.304389    4576 scope.go:122] "RemoveContainer" containerID="7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f"
Oct 30 01:24:15.324400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.324377    4576 scope.go:122] "RemoveContainer" containerID="7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f"
Oct 30 01:24:15.324703 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:15.324675    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f\": container with ID starting with 7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f not found: ID does not exist" containerID="7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f"
Oct 30 01:24:15.324818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.324717    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f"} err="failed to get container status \"7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f\": rpc error: code = NotFound desc = could not find container \"7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f\": container with ID starting with 7c9aa2dac4c7940bc1b777b76ee16a8b13c802e198776db05f0d0fb9e6b8e10f not found: ID does not exist"
Oct 30 01:24:15.366878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.366849    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"]
Oct 30 01:24:15.372637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.372604    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3c2596-openshift-default-6b68bd79df-fckft"]
Oct 30 01:24:15.374113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:15.374049    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:16.046378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:16.046339    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:16.109304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:16.109274    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" path="/var/lib/kubelet/pods/f0987db5-792d-4300-a535-2dd9b7a17cb3/volumes"
Oct 30 01:24:16.267738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:16.267692    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:16.278447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:16.278418    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"]
Oct 30 01:24:16.291532 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:16.291506    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b05110-ff78f"]
Oct 30 01:24:16.415405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:16.415304    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:16.746037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:16.745999    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:17.028738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:17.028648    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"]
Oct 30 01:24:17.042374 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:17.042348    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e424f59-txv5c"]
Oct 30 01:24:17.395151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:17.395067    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:17.531073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:17.531039    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:18.109488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:18.109453    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c39a658-acf1-4334-8f1f-12fa46ce65a5" path="/var/lib/kubelet/pods/2c39a658-acf1-4334-8f1f-12fa46ce65a5/volumes"
Oct 30 01:24:18.109925 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:18.109910    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8a880a-e9e2-4b92-9497-4fb7492a1bb2" path="/var/lib/kubelet/pods/4b8a880a-e9e2-4b92-9497-4fb7492a1bb2/volumes"
Oct 30 01:24:18.383089 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:18.383009    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:24:18.553256 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:18.553198    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"]
Oct 30 01:24:18.553717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:18.553461    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8" podUID="f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" containerName="maas-api" containerID="cri-o://9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84" gracePeriod=30
Oct 30 01:24:18.622552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:18.622509    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:18.958966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:18.958634    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 01:24:19.084267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.050812    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-openshift-service-ca\") pod \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") "
Oct 30 01:24:19.089297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.072208    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:19.093386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.077711    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-openshift-service-ca" pod "f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" (UID: "f8d892ed-cfd0-4c31-b52e-c11f800cd7ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:19.100297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.089599    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-kube-api-access-w94qs\" (UniqueName: \"kubernetes.io/projected/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-kube-api-access-w94qs\") pod \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") "
Oct 30 01:24:19.100536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.100516    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-maas-api-tls\") pod \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") "
Oct 30 01:24:19.115253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.109364    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-metrics-certs\") pod \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\" (UID: \"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce\") "
Oct 30 01:24:19.125585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.120585    4576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-openshift-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.125585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.116165    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-kube-api-access-w94qs" pod "f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" (UID: "f8d892ed-cfd0-4c31-b52e-c11f800cd7ce"). InnerVolumeSpecName "kube-api-access-w94qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:19.125585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.119563    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-maas-api-tls" pod "f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" (UID: "f8d892ed-cfd0-4c31-b52e-c11f800cd7ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:19.125585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.121044    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-metrics-certs" pod "f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" (UID: "f8d892ed-cfd0-4c31-b52e-c11f800cd7ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:19.211715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.211625    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:19.222069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.222030    4576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-metrics-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.222069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.222058    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w94qs\" (UniqueName: \"kubernetes.io/projected/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-kube-api-access-w94qs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.222069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.222070    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce-maas-api-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.319708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.319672    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:19.339391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.339357    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"]
Oct 30 01:24:19.342994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.342972    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca324d-rshs2"]
Oct 30 01:24:19.343145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.343008    4576 generic.go:356] "Generic (PLEG): container finished" podID="f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" containerID="9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84" exitCode=0
Oct 30 01:24:19.343145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.343063    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"
Oct 30 01:24:19.343145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.343012    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8" event={"ID":"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce","Type":"ContainerDied","Data":"9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84"}
Oct 30 01:24:19.343341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.343170    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8" event={"ID":"f8d892ed-cfd0-4c31-b52e-c11f800cd7ce","Type":"ContainerDied","Data":"f766683e72e0c187b23bdadb791cbdd627a5ca18fe4ca6022decae30b3d545f0"}
Oct 30 01:24:19.343341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.343205    4576 scope.go:122] "RemoveContainer" containerID="9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84"
Oct 30 01:24:19.355002 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.354984    4576 scope.go:122] "RemoveContainer" containerID="9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84"
Oct 30 01:24:19.355283 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:19.355259    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84\": container with ID starting with 9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84 not found: ID does not exist" containerID="9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84"
Oct 30 01:24:19.355388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.355289    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84"} err="failed to get container status \"9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84\": rpc error: code = NotFound desc = could not find container \"9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84\": container with ID starting with 9bb0e4154709cebd90d394dd3222c34b2ddd9376566d7e7065ce63ac0f5f5b84 not found: ID does not exist"
Oct 30 01:24:19.358795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.358773    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff" podUID="d50140ed-d37d-4110-8331-71bf08574258" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:19.402676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.402641    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"]
Oct 30 01:24:19.406078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.406055    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ca324d-6c68788c55-vr6l8"]
Oct 30 01:24:19.703382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.703343    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:19.749322 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.749284    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6" podUID="641d53d4-0a3c-496d-898d-bbc381ba8102" containerName="payload-processing" containerID="cri-o://d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe" gracePeriod=25
Oct 30 01:24:19.786638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.786602    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm" podUID="57f55c7e-f518-4485-a436-fa6181614bdf" containerName="payload-pre-processing" containerID="cri-o://c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38" gracePeriod=25
Oct 30 01:24:19.825372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:19.825336    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:20.056592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:20.056504    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:20.084177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:20.084149    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"]
Oct 30 01:24:20.087406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:20.087386    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-42d56b-cz5m2"]
Oct 30 01:24:20.109609 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:20.109580    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6" path="/var/lib/kubelet/pods/3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6/volumes"
Oct 30 01:24:20.110084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:20.110068    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699f02fd-8493-4143-983c-6b8767fcd27a" path="/var/lib/kubelet/pods/699f02fd-8493-4143-983c-6b8767fcd27a/volumes"
Oct 30 01:24:20.110563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:20.110548    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" path="/var/lib/kubelet/pods/f8d892ed-cfd0-4c31-b52e-c11f800cd7ce/volumes"
Oct 30 01:24:20.170549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:20.170478    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:20.570462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:20.570433    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:21.315379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:21.315298    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:21.429020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:21.428982    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:21.880717 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:21.880682    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:22.657703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:22.657671    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:22.767555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:22.767523    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:22.978269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:22.977791    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"]
Oct 30 01:24:22.978269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:22.978046    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy" containerID="cri-o://57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1" gracePeriod=30
Oct 30 01:24:23.033306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.033208    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:23.884729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.884697    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"]
Oct 30 01:24:23.885316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885299    4576 cpu_manager.go:414] "containerMap: removing container" podUID="fdc83f61-3437-423b-b0e6-096c8f017880" containerName="main"
Oct 30 01:24:23.885316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885315    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc83f61-3437-423b-b0e6-096c8f017880" containerName="main"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885328    4576 cpu_manager.go:414] "containerMap: removing container" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885333    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885342    4576 cpu_manager.go:414] "containerMap: removing container" podUID="92ab0966-c954-43f1-8478-d31e88703b0e" containerName="revoke-keys"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885350    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ab0966-c954-43f1-8478-d31e88703b0e" containerName="revoke-keys"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885362    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" containerName="maas-api"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885367    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" containerName="maas-api"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885392    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885404    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885421    4576 cpu_manager.go:414] "containerMap: removing container" podUID="3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6" containerName="revoke-keys"
Oct 30 01:24:23.885478 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885425    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6" containerName="revoke-keys"
Oct 30 01:24:23.885788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885529    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0987db5-792d-4300-a535-2dd9b7a17cb3" containerName="istio-proxy"
Oct 30 01:24:23.885788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885539    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be111cd-3a03-4ff0-bcf7-f5bc59ecb7e6" containerName="revoke-keys"
Oct 30 01:24:23.885788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885548    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc83f61-3437-423b-b0e6-096c8f017880" containerName="main"
Oct 30 01:24:23.885788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885553    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="900b3754-1cef-4071-b077-2663fcd55336" containerName="istio-proxy"
Oct 30 01:24:23.885788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885558    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ab0966-c954-43f1-8478-d31e88703b0e" containerName="revoke-keys"
Oct 30 01:24:23.885788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.885562    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d892ed-cfd0-4c31-b52e-c11f800cd7ce" containerName="maas-api"
Oct 30 01:24:23.890526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.890507    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:23.906567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.906540    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:23.906840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.906823    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"]
Oct 30 01:24:23.915701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.915675    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:24:23.935090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.935060    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:23.947273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.947245    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446406f2-5194-45db-bd67-da815db637b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:23.947394 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.947368    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5km\" (UniqueName: \"kubernetes.io/projected/446406f2-5194-45db-bd67-da815db637b5-kube-api-access-7p5km\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:23.947524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:23.947510    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446406f2-5194-45db-bd67-da815db637b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:24.049007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:24.048952    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446406f2-5194-45db-bd67-da815db637b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:24.049197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:24.049066    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5km\" (UniqueName: \"kubernetes.io/projected/446406f2-5194-45db-bd67-da815db637b5-kube-api-access-7p5km\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:24.049283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:24.049202    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446406f2-5194-45db-bd67-da815db637b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:24.049682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:24.049661    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446406f2-5194-45db-bd67-da815db637b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:24.051699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:24.051680    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446406f2-5194-45db-bd67-da815db637b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:24.069571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:24.069542    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5km\" (UniqueName: \"kubernetes.io/projected/446406f2-5194-45db-bd67-da815db637b5-kube-api-access-7p5km\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:24.208776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:24.208747    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:24.512055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:24.512025    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"]
Oct 30 01:24:24.517322 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:24.517293    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446406f2_5194_45db_bd67_da815db637b5.slice/crio-07075ce08496c8dee544336d1d5abd9e30baaf7ce93bf8334cf63f4e4127a59e WatchSource:0}: Error finding container 07075ce08496c8dee544336d1d5abd9e30baaf7ce93bf8334cf63f4e4127a59e: Status 404 returned error can't find the container with id 07075ce08496c8dee544336d1d5abd9e30baaf7ce93bf8334cf63f4e4127a59e
Oct 30 01:24:25.393253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:25.393183    4576 generic.go:356] "Generic (PLEG): container finished" podID="446406f2-5194-45db-bd67-da815db637b5" containerID="a1b3ce2902a2b0ff7f813367bccda659d18523f395f2899cb71b52b78cdf5104" exitCode=0
Oct 30 01:24:25.393647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:25.393263    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8" event={"ID":"446406f2-5194-45db-bd67-da815db637b5","Type":"ContainerDied","Data":"a1b3ce2902a2b0ff7f813367bccda659d18523f395f2899cb71b52b78cdf5104"}
Oct 30 01:24:25.393647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:25.393348    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8" event={"ID":"446406f2-5194-45db-bd67-da815db637b5","Type":"ContainerStarted","Data":"07075ce08496c8dee544336d1d5abd9e30baaf7ce93bf8334cf63f4e4127a59e"}
Oct 30 01:24:25.652875 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:25.652763    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:25.773779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:25.773744    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:25.948864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:25.948824    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm" podUID="57f55c7e-f518-4485-a436-fa6181614bdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:25.949082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:25.948824    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6" podUID="641d53d4-0a3c-496d-898d-bbc381ba8102" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:26.016490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.016457    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:26.486069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.486042    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"]
Oct 30 01:24:26.491392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.491364    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.521289 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.513153    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"]
Oct 30 01:24:26.566529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.566502    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:26.588501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.588474    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.588656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.588641    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7nt\" (UniqueName: \"kubernetes.io/projected/57d9019e-8a2b-4d5d-9ef5-e6a457805992-kube-api-access-rz7nt\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.588769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.588753    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.653598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.653458    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:26.689579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.689548    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/446406f2-5194-45db-bd67-da815db637b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446406f2-5194-45db-bd67-da815db637b5-maas-api-openshift-ca\") pod \"446406f2-5194-45db-bd67-da815db637b5\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") "
Oct 30 01:24:26.689755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.689629    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446406f2-5194-45db-bd67-da815db637b5-kube-api-access-7p5km\" (UniqueName: \"kubernetes.io/projected/446406f2-5194-45db-bd67-da815db637b5-kube-api-access-7p5km\") pod \"446406f2-5194-45db-bd67-da815db637b5\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") "
Oct 30 01:24:26.689755 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.689665    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/446406f2-5194-45db-bd67-da815db637b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446406f2-5194-45db-bd67-da815db637b5-maas-api-xks-ca\") pod \"446406f2-5194-45db-bd67-da815db637b5\" (UID: \"446406f2-5194-45db-bd67-da815db637b5\") "
Oct 30 01:24:26.689899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.689875    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446406f2-5194-45db-bd67-da815db637b5-maas-api-openshift-ca" pod "446406f2-5194-45db-bd67-da815db637b5" (UID: "446406f2-5194-45db-bd67-da815db637b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:26.689966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.689889    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.690050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.690032    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.690175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.690160    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7nt\" (UniqueName: \"kubernetes.io/projected/57d9019e-8a2b-4d5d-9ef5-e6a457805992-kube-api-access-rz7nt\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.690305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.690289    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446406f2-5194-45db-bd67-da815db637b5-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.690686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.690666    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.692059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.692031    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446406f2-5194-45db-bd67-da815db637b5-maas-api-xks-ca" pod "446406f2-5194-45db-bd67-da815db637b5" (UID: "446406f2-5194-45db-bd67-da815db637b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:26.692213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.692150    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446406f2-5194-45db-bd67-da815db637b5-kube-api-access-7p5km" pod "446406f2-5194-45db-bd67-da815db637b5" (UID: "446406f2-5194-45db-bd67-da815db637b5"). InnerVolumeSpecName "kube-api-access-7p5km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:26.692698 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.692672    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.711819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.711789    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7nt\" (UniqueName: \"kubernetes.io/projected/57d9019e-8a2b-4d5d-9ef5-e6a457805992-kube-api-access-rz7nt\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.766403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.766370    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:26.791394 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.791348    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p5km\" (UniqueName: \"kubernetes.io/projected/446406f2-5194-45db-bd67-da815db637b5-kube-api-access-7p5km\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.791394 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.791392    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446406f2-5194-45db-bd67-da815db637b5-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:26.816337 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.816310    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:26.927031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.926941    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"]
Oct 30 01:24:26.927223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.927180    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy" containerID="cri-o://be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b" gracePeriod=30
Oct 30 01:24:26.975145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:26.975064    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:27.040840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.039314    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"]
Oct 30 01:24:27.040840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.039628    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26d9a2c9-pf6rm"]
Oct 30 01:24:27.149689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.149594    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"]
Oct 30 01:24:27.151541 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:27.151512    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d9019e_8a2b_4d5d_9ef5_e6a457805992.slice/crio-17c45b3c2713848ccd239434c219371d8a7c1a8f53cdbb418fec4415b4ae3f99 WatchSource:0}: Error finding container 17c45b3c2713848ccd239434c219371d8a7c1a8f53cdbb418fec4415b4ae3f99: Status 404 returned error can't find the container with id 17c45b3c2713848ccd239434c219371d8a7c1a8f53cdbb418fec4415b4ae3f99
Oct 30 01:24:27.410386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.410346    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j" event={"ID":"57d9019e-8a2b-4d5d-9ef5-e6a457805992","Type":"ContainerStarted","Data":"17c45b3c2713848ccd239434c219371d8a7c1a8f53cdbb418fec4415b4ae3f99"}
Oct 30 01:24:27.411776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.411753    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"
Oct 30 01:24:27.411904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.411793    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8" event={"ID":"446406f2-5194-45db-bd67-da815db637b5","Type":"ContainerDied","Data":"07075ce08496c8dee544336d1d5abd9e30baaf7ce93bf8334cf63f4e4127a59e"}
Oct 30 01:24:27.411904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.411843    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07075ce08496c8dee544336d1d5abd9e30baaf7ce93bf8334cf63f4e4127a59e"
Oct 30 01:24:27.419986 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.419969    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"]
Oct 30 01:24:27.420617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.420599    4576 cpu_manager.go:414] "containerMap: removing container" podUID="446406f2-5194-45db-bd67-da815db637b5" containerName="revoke-keys"
Oct 30 01:24:27.420617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.420614    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446406f2-5194-45db-bd67-da815db637b5" containerName="revoke-keys"
Oct 30 01:24:27.420753 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.420729    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="446406f2-5194-45db-bd67-da815db637b5" containerName="revoke-keys"
Oct 30 01:24:27.425409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.425385    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.440859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.440807    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"]
Oct 30 01:24:27.539145 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.539097    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.539657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.539251    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rkdq\" (UniqueName: \"kubernetes.io/projected/403dab46-3331-4dc6-a51e-a089c59b9d0d-kube-api-access-2rkdq\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.539657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.539335    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.615013 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.614986    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:27.640348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.640309    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.640567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.640409    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkdq\" (UniqueName: \"kubernetes.io/projected/403dab46-3331-4dc6-a51e-a089c59b9d0d-kube-api-access-2rkdq\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.640567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.640506    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.641269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.641170    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.643679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.643651    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.666419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.666389    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkdq\" (UniqueName: \"kubernetes.io/projected/403dab46-3331-4dc6-a51e-a089c59b9d0d-kube-api-access-2rkdq\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:27.739018 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:27.738937    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:28.045848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.045823    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"]
Oct 30 01:24:28.070923 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:28.070890    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403dab46_3331_4dc6_a51e_a089c59b9d0d.slice/crio-0c43eef26eb2cdd5461a08cab8aa2d2c28bd15e37abc7e1bff9d98832ff09425 WatchSource:0}: Error finding container 0c43eef26eb2cdd5461a08cab8aa2d2c28bd15e37abc7e1bff9d98832ff09425: Status 404 returned error can't find the container with id 0c43eef26eb2cdd5461a08cab8aa2d2c28bd15e37abc7e1bff9d98832ff09425
Oct 30 01:24:28.109611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.109580    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c55f2bc-6e5e-46b3-88ad-70a48c525dfa" path="/var/lib/kubelet/pods/3c55f2bc-6e5e-46b3-88ad-70a48c525dfa/volumes"
Oct 30 01:24:28.241795 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.241771    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:24:28.349617 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.349581    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-kube-api-access-s5jrn\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-kube-api-access-s5jrn\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.349806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.349660    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-data\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.349806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.349700    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-envoy\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.349806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.349752    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-socket\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.349955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.349873    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-credential-socket\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.349955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.349932    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-certs\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.350063 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.350024    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istiod-ca-cert\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.350114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.350072    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-podinfo\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.350165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.350135    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-token\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-token\") pod \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\" (UID: \"a5ba917c-f190-46e5-9a6a-094f37d98c6a\") "
Oct 30 01:24:28.351051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.350343    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-socket" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:28.351051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.350840    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.351619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.351353    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-data" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:28.352934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.352908    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-kube-api-access-s5jrn" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "kube-api-access-s5jrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:28.353159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.353136    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-certs" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:28.353384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.353359    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-credential-socket" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:28.353644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.353626    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istiod-ca-cert" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:28.354304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.354276    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-token" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:28.354628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.354603    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-envoy" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:28.355921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.355901    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-podinfo" pod "a5ba917c-f190-46e5-9a6a-094f37d98c6a" (UID: "a5ba917c-f190-46e5-9a6a-094f37d98c6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:28.421575 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.421537    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8" event={"ID":"403dab46-3331-4dc6-a51e-a089c59b9d0d","Type":"ContainerStarted","Data":"69382a9f189457e6742f0ce1911feb44a319237d5bcbc81cf8d2cf6b885d74c4"}
Oct 30 01:24:28.421752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.421602    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8" event={"ID":"403dab46-3331-4dc6-a51e-a089c59b9d0d","Type":"ContainerStarted","Data":"0c43eef26eb2cdd5461a08cab8aa2d2c28bd15e37abc7e1bff9d98832ff09425"}
Oct 30 01:24:28.423064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.423037    4576 generic.go:356] "Generic (PLEG): container finished" podID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerID="57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1" exitCode=0
Oct 30 01:24:28.423202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.423074    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"
Oct 30 01:24:28.423202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.423113    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" event={"ID":"a5ba917c-f190-46e5-9a6a-094f37d98c6a","Type":"ContainerDied","Data":"57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1"}
Oct 30 01:24:28.423202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.423160    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd" event={"ID":"a5ba917c-f190-46e5-9a6a-094f37d98c6a","Type":"ContainerDied","Data":"73a69d5283d758d1383fbf01ff82096e2061a0c974aef5faf51a78ae176e0a2d"}
Oct 30 01:24:28.423202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.423181    4576 scope.go:122] "RemoveContainer" containerID="57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1"
Oct 30 01:24:28.424874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.424854    4576 generic.go:356] "Generic (PLEG): container finished" podID="57d9019e-8a2b-4d5d-9ef5-e6a457805992" containerID="18d427254fe1adf2592333e7704db3d06dcda0482cd8315b7909353f1797dbf6" exitCode=0
Oct 30 01:24:28.424985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.424931    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j" event={"ID":"57d9019e-8a2b-4d5d-9ef5-e6a457805992","Type":"ContainerDied","Data":"18d427254fe1adf2592333e7704db3d06dcda0482cd8315b7909353f1797dbf6"}
Oct 30 01:24:28.434884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.434867    4576 scope.go:122] "RemoveContainer" containerID="57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1"
Oct 30 01:24:28.435141 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:28.435123    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1\": container with ID starting with 57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1 not found: ID does not exist" containerID="57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1"
Oct 30 01:24:28.435190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.435150    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1"} err="failed to get container status \"57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1\": rpc error: code = NotFound desc = could not find container \"57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1\": container with ID starting with 57a21ebd2075dc0cd34467aaa333a8e97db7f77ea10d3119ae0e2da569cfcde1 not found: ID does not exist"
Oct 30 01:24:28.452412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.452383    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.452412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.452409    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.452412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.452417    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.452611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.452426    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jrn\" (UniqueName: \"kubernetes.io/projected/a5ba917c-f190-46e5-9a6a-094f37d98c6a-kube-api-access-s5jrn\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.452611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.452435    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.452611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.452445    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.452611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.452452    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.452611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.452460    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ba917c-f190-46e5-9a6a-094f37d98c6a-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:28.467437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.467381    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8" podStartSLOduration=1.4673580689999999 podStartE2EDuration="1.467358069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:28.460834667 +0000 UTC m=+2748.532158513" watchObservedRunningTime="2026-10-30 01:24:28.467358069 +0000 UTC m=+2748.538681905"
Oct 30 01:24:28.502356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.502327    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"]
Oct 30 01:24:28.505442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.505418    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-b05110-openshift-default-654b7457b7-7l4sd"]
Oct 30 01:24:28.862365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.862282    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"]
Oct 30 01:24:28.897036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:28.897006    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"]
Oct 30 01:24:29.024700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.024667    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:29.139195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.139114    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:29.242535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.242446    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:29.358907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.358867    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff" podUID="d50140ed-d37d-4110-8331-71bf08574258" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:29.434387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.434303    4576 generic.go:356] "Generic (PLEG): container finished" podID="403dab46-3331-4dc6-a51e-a089c59b9d0d" containerID="69382a9f189457e6742f0ce1911feb44a319237d5bcbc81cf8d2cf6b885d74c4" exitCode=0
Oct 30 01:24:29.434558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.434387    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8" event={"ID":"403dab46-3331-4dc6-a51e-a089c59b9d0d","Type":"ContainerDied","Data":"69382a9f189457e6742f0ce1911feb44a319237d5bcbc81cf8d2cf6b885d74c4"}
Oct 30 01:24:29.573458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.573418    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:29.598603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.598567    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:29.775326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.775292    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d9019e-8a2b-4d5d-9ef5-e6a457805992-kube-api-access-rz7nt\" (UniqueName: \"kubernetes.io/projected/57d9019e-8a2b-4d5d-9ef5-e6a457805992-kube-api-access-rz7nt\") pod \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") "
Oct 30 01:24:29.775567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.775357    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-openshift-ca\") pod \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") "
Oct 30 01:24:29.775567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.775478    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-xks-ca\") pod \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\" (UID: \"57d9019e-8a2b-4d5d-9ef5-e6a457805992\") "
Oct 30 01:24:29.775841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.775813    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-openshift-ca" pod "57d9019e-8a2b-4d5d-9ef5-e6a457805992" (UID: "57d9019e-8a2b-4d5d-9ef5-e6a457805992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:29.776052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.776032    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.777758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.777729    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-xks-ca" pod "57d9019e-8a2b-4d5d-9ef5-e6a457805992" (UID: "57d9019e-8a2b-4d5d-9ef5-e6a457805992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:29.777856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.777837    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d9019e-8a2b-4d5d-9ef5-e6a457805992-kube-api-access-rz7nt" pod "57d9019e-8a2b-4d5d-9ef5-e6a457805992" (UID: "57d9019e-8a2b-4d5d-9ef5-e6a457805992"). InnerVolumeSpecName "kube-api-access-rz7nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:29.877673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.877633    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz7nt\" (UniqueName: \"kubernetes.io/projected/57d9019e-8a2b-4d5d-9ef5-e6a457805992-kube-api-access-rz7nt\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.877673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:29.877672    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d9019e-8a2b-4d5d-9ef5-e6a457805992-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:30.109011 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.108922    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" path="/var/lib/kubelet/pods/a5ba917c-f190-46e5-9a6a-094f37d98c6a/volumes"
Oct 30 01:24:30.444649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.444617    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"
Oct 30 01:24:30.444847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.444657    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j" event={"ID":"57d9019e-8a2b-4d5d-9ef5-e6a457805992","Type":"ContainerDied","Data":"17c45b3c2713848ccd239434c219371d8a7c1a8f53cdbb418fec4415b4ae3f99"}
Oct 30 01:24:30.444847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.444711    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c45b3c2713848ccd239434c219371d8a7c1a8f53cdbb418fec4415b4ae3f99"
Oct 30 01:24:30.601200 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.601172    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:30.688552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.688522    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403dab46-3331-4dc6-a51e-a089c59b9d0d-kube-api-access-2rkdq\" (UniqueName: \"kubernetes.io/projected/403dab46-3331-4dc6-a51e-a089c59b9d0d-kube-api-access-2rkdq\") pod \"403dab46-3331-4dc6-a51e-a089c59b9d0d\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") "
Oct 30 01:24:30.688742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.688619    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-xks-ca\") pod \"403dab46-3331-4dc6-a51e-a089c59b9d0d\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") "
Oct 30 01:24:30.688742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.688645    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-openshift-ca\") pod \"403dab46-3331-4dc6-a51e-a089c59b9d0d\" (UID: \"403dab46-3331-4dc6-a51e-a089c59b9d0d\") "
Oct 30 01:24:30.689142 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.689101    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-openshift-ca" pod "403dab46-3331-4dc6-a51e-a089c59b9d0d" (UID: "403dab46-3331-4dc6-a51e-a089c59b9d0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:30.689462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.689267    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:30.690890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.690862    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-xks-ca" pod "403dab46-3331-4dc6-a51e-a089c59b9d0d" (UID: "403dab46-3331-4dc6-a51e-a089c59b9d0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:30.691047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.691025    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403dab46-3331-4dc6-a51e-a089c59b9d0d-kube-api-access-2rkdq" pod "403dab46-3331-4dc6-a51e-a089c59b9d0d" (UID: "403dab46-3331-4dc6-a51e-a089c59b9d0d"). InnerVolumeSpecName "kube-api-access-2rkdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:30.790760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.790724    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403dab46-3331-4dc6-a51e-a089c59b9d0d-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:30.790760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:30.790759    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rkdq\" (UniqueName: \"kubernetes.io/projected/403dab46-3331-4dc6-a51e-a089c59b9d0d-kube-api-access-2rkdq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:31.384616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:31.384572    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:31.454506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:31.454478    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"
Oct 30 01:24:31.454710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:31.454513    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8" event={"ID":"403dab46-3331-4dc6-a51e-a089c59b9d0d","Type":"ContainerDied","Data":"0c43eef26eb2cdd5461a08cab8aa2d2c28bd15e37abc7e1bff9d98832ff09425"}
Oct 30 01:24:31.454710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:31.454565    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c43eef26eb2cdd5461a08cab8aa2d2c28bd15e37abc7e1bff9d98832ff09425"
Oct 30 01:24:32.234164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.234140    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:24:32.410116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410080    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-kube-api-access-fgdpl\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-kube-api-access-fgdpl\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410166    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-credential-socket\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410201    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-socket\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410248    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istiod-ca-cert\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410297    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-podinfo\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410362    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-envoy\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410418    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-data\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410473    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-token\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-token\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410541    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-socket" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410590    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-certs\") pod \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\" (UID: \"8aaa2da1-a942-426f-8fb5-79d3d5be7b84\") "
Oct 30 01:24:32.410665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410628    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-credential-socket" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:32.411273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410708    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istiod-ca-cert" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:32.411273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410854    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-data" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:32.411273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.410926    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-certs" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:32.411505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.411475    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.411636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.411508    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.411636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.411525    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.411636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.411539    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.411636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.411555    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.412751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.412731    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-kube-api-access-fgdpl" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "kube-api-access-fgdpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:32.413348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.413330    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-podinfo" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:32.413546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.413517    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-envoy" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:32.413546 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.413531    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-token" pod "8aaa2da1-a942-426f-8fb5-79d3d5be7b84" (UID: "8aaa2da1-a942-426f-8fb5-79d3d5be7b84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:32.463534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.463455    4576 generic.go:356] "Generic (PLEG): container finished" podID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerID="be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b" exitCode=0
Oct 30 01:24:32.463534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.463510    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" event={"ID":"8aaa2da1-a942-426f-8fb5-79d3d5be7b84","Type":"ContainerDied","Data":"be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b"}
Oct 30 01:24:32.463534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.463515    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"
Oct 30 01:24:32.463751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.463580    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h" event={"ID":"8aaa2da1-a942-426f-8fb5-79d3d5be7b84","Type":"ContainerDied","Data":"5a365acbfcd3bb72d7b51ea036d9e9d1d229d469339f67f66c03eeb0dd4201f5"}
Oct 30 01:24:32.463751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.463611    4576 scope.go:122] "RemoveContainer" containerID="be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b"
Oct 30 01:24:32.475422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.475406    4576 scope.go:122] "RemoveContainer" containerID="be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b"
Oct 30 01:24:32.475705 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:32.475683    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b\": container with ID starting with be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b not found: ID does not exist" containerID="be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b"
Oct 30 01:24:32.475750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.475715    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b"} err="failed to get container status \"be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b\": rpc error: code = NotFound desc = could not find container \"be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b\": container with ID starting with be4bd95fad934375b8091b56e54d2897f62177516163f621942276f0ac8b301b not found: ID does not exist"
Oct 30 01:24:32.512498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.512466    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgdpl\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-kube-api-access-fgdpl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.512498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.512500    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.512498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.512510    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.512696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.512519    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aaa2da1-a942-426f-8fb5-79d3d5be7b84-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:32.522066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.522041    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"]
Oct 30 01:24:32.525901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:32.525873    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-42d56b-openshift-default-658c44cc97-rwt4h"]
Oct 30 01:24:33.022314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:33.022285    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:33.157259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:33.150199    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:33.866259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:33.865315    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt" podUID="00d9aef7-2d8f-4e94-b364-82d203e38739" containerName="payload-processing" containerID="cri-o://d3ad6c42b2165fe5ae3fdf5ab00eb7d6b8dc18dc630e07f703149a1f12001f13" gracePeriod=25
Oct 30 01:24:33.907269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:33.899312    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn" podUID="ec04e918-bc46-4d4f-b2fa-735b886ea3b5" containerName="payload-pre-processing" containerID="cri-o://bb97ca754f09fb217c3f31a2bdb60cf861d525d5e7bce2338cbc8a02003d3822" gracePeriod=25
Oct 30 01:24:34.134309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.129387    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" path="/var/lib/kubelet/pods/8aaa2da1-a942-426f-8fb5-79d3d5be7b84/volumes"
Oct 30 01:24:34.554620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.547930    4576 generic.go:356] "Generic (PLEG): container finished" podID="d50140ed-d37d-4110-8331-71bf08574258" containerID="c93f6425dfad5b07022ec4f1a8178f7ca4998ce4f44bc13459260d866e76576a" exitCode=0
Oct 30 01:24:34.554620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.548004    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff" event={"ID":"d50140ed-d37d-4110-8331-71bf08574258","Type":"ContainerDied","Data":"c93f6425dfad5b07022ec4f1a8178f7ca4998ce4f44bc13459260d866e76576a"}
Oct 30 01:24:34.554620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.550586    4576 generic.go:356] "Generic (PLEG): container finished" podID="00d9aef7-2d8f-4e94-b364-82d203e38739" containerID="d3ad6c42b2165fe5ae3fdf5ab00eb7d6b8dc18dc630e07f703149a1f12001f13" exitCode=0
Oct 30 01:24:34.554620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.550861    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt" event={"ID":"00d9aef7-2d8f-4e94-b364-82d203e38739","Type":"ContainerDied","Data":"d3ad6c42b2165fe5ae3fdf5ab00eb7d6b8dc18dc630e07f703149a1f12001f13"}
Oct 30 01:24:34.554620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.553115    4576 generic.go:356] "Generic (PLEG): container finished" podID="ec04e918-bc46-4d4f-b2fa-735b886ea3b5" containerID="bb97ca754f09fb217c3f31a2bdb60cf861d525d5e7bce2338cbc8a02003d3822" exitCode=0
Oct 30 01:24:34.554620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.553283    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn" event={"ID":"ec04e918-bc46-4d4f-b2fa-735b886ea3b5","Type":"ContainerDied","Data":"bb97ca754f09fb217c3f31a2bdb60cf861d525d5e7bce2338cbc8a02003d3822"}
Oct 30 01:24:34.579251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.579214    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:24:34.680590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.680556    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:24:34.693405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.693364    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d9aef7-2d8f-4e94-b364-82d203e38739-kube-api-access-nt56t\" (UniqueName: \"kubernetes.io/projected/00d9aef7-2d8f-4e94-b364-82d203e38739-kube-api-access-nt56t\") pod \"00d9aef7-2d8f-4e94-b364-82d203e38739\" (UID: \"00d9aef7-2d8f-4e94-b364-82d203e38739\") "
Oct 30 01:24:34.693529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.693430    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d9aef7-2d8f-4e94-b364-82d203e38739-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d9aef7-2d8f-4e94-b364-82d203e38739-plugins-config-volume\") pod \"00d9aef7-2d8f-4e94-b364-82d203e38739\" (UID: \"00d9aef7-2d8f-4e94-b364-82d203e38739\") "
Oct 30 01:24:34.693843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.693819    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d9aef7-2d8f-4e94-b364-82d203e38739-plugins-config-volume" pod "00d9aef7-2d8f-4e94-b364-82d203e38739" (UID: "00d9aef7-2d8f-4e94-b364-82d203e38739"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:34.693968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.693945    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d9aef7-2d8f-4e94-b364-82d203e38739-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:34.695642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.695621    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d9aef7-2d8f-4e94-b364-82d203e38739-kube-api-access-nt56t" pod "00d9aef7-2d8f-4e94-b364-82d203e38739" (UID: "00d9aef7-2d8f-4e94-b364-82d203e38739"). InnerVolumeSpecName "kube-api-access-nt56t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:34.707019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.706993    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:24:34.779394 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.779360    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:34.795109 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.795080    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50140ed-d37d-4110-8331-71bf08574258-kube-api-access-sb76g\" (UniqueName: \"kubernetes.io/projected/d50140ed-d37d-4110-8331-71bf08574258-kube-api-access-sb76g\") pod \"d50140ed-d37d-4110-8331-71bf08574258\" (UID: \"d50140ed-d37d-4110-8331-71bf08574258\") "
Oct 30 01:24:34.795353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.795127    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-plugins-config-volume\") pod \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\" (UID: \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\") "
Oct 30 01:24:34.795353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.795175    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-kube-api-access-2ndss\" (UniqueName: \"kubernetes.io/projected/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-kube-api-access-2ndss\") pod \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\" (UID: \"ec04e918-bc46-4d4f-b2fa-735b886ea3b5\") "
Oct 30 01:24:34.795507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.795387    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50140ed-d37d-4110-8331-71bf08574258-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50140ed-d37d-4110-8331-71bf08574258-plugins-config-volume\") pod \"d50140ed-d37d-4110-8331-71bf08574258\" (UID: \"d50140ed-d37d-4110-8331-71bf08574258\") "
Oct 30 01:24:34.795885 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.795696    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-plugins-config-volume" pod "ec04e918-bc46-4d4f-b2fa-735b886ea3b5" (UID: "ec04e918-bc46-4d4f-b2fa-735b886ea3b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:34.795885 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.795743    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50140ed-d37d-4110-8331-71bf08574258-plugins-config-volume" pod "d50140ed-d37d-4110-8331-71bf08574258" (UID: "d50140ed-d37d-4110-8331-71bf08574258"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:34.796709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.796685    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt56t\" (UniqueName: \"kubernetes.io/projected/00d9aef7-2d8f-4e94-b364-82d203e38739-kube-api-access-nt56t\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:34.796709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.796715    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:34.796877 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.796729    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50140ed-d37d-4110-8331-71bf08574258-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:34.797754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.797725    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-kube-api-access-2ndss" pod "ec04e918-bc46-4d4f-b2fa-735b886ea3b5" (UID: "ec04e918-bc46-4d4f-b2fa-735b886ea3b5"). InnerVolumeSpecName "kube-api-access-2ndss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:34.798060 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.798030    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50140ed-d37d-4110-8331-71bf08574258-kube-api-access-sb76g" pod "d50140ed-d37d-4110-8331-71bf08574258" (UID: "d50140ed-d37d-4110-8331-71bf08574258"). InnerVolumeSpecName "kube-api-access-sb76g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:34.902277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.900763    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:34.902277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.900786    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndss\" (UniqueName: \"kubernetes.io/projected/ec04e918-bc46-4d4f-b2fa-735b886ea3b5-kube-api-access-2ndss\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:34.902277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:34.900806    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb76g\" (UniqueName: \"kubernetes.io/projected/d50140ed-d37d-4110-8331-71bf08574258-kube-api-access-sb76g\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:35.102460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.102426    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:35.562423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.562392    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"
Oct 30 01:24:35.562612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.562384    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff" event={"ID":"d50140ed-d37d-4110-8331-71bf08574258","Type":"ContainerDied","Data":"ac9027db06352e04ae81b2d6314b8d0762097a7481897177ac9e2ce9f67a66f1"}
Oct 30 01:24:35.562612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.562551    4576 scope.go:122] "RemoveContainer" containerID="c93f6425dfad5b07022ec4f1a8178f7ca4998ce4f44bc13459260d866e76576a"
Oct 30 01:24:35.564388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.564364    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"
Oct 30 01:24:35.564388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.564379    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt" event={"ID":"00d9aef7-2d8f-4e94-b364-82d203e38739","Type":"ContainerDied","Data":"cdd960fc557439dc9889b154fa864be37ec41ebd2b576f2f388b5c278864d9bc"}
Oct 30 01:24:35.566273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.566252    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"
Oct 30 01:24:35.566273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.566260    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn" event={"ID":"ec04e918-bc46-4d4f-b2fa-735b886ea3b5","Type":"ContainerDied","Data":"aecbe25bec883bcdec3200d0aeab1a613fc86e493b80551551e656797c8001db"}
Oct 30 01:24:35.603752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.603728    4576 scope.go:122] "RemoveContainer" containerID="d3ad6c42b2165fe5ae3fdf5ab00eb7d6b8dc18dc630e07f703149a1f12001f13"
Oct 30 01:24:35.615968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.615948    4576 scope.go:122] "RemoveContainer" containerID="bb97ca754f09fb217c3f31a2bdb60cf861d525d5e7bce2338cbc8a02003d3822"
Oct 30 01:24:35.659917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.659887    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"]
Oct 30 01:24:35.696741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.696699    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-b05110-68f74d7776-stfff"]
Oct 30 01:24:35.709824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.709785    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"]
Oct 30 01:24:35.713282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.713224    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-11e7a9-7bb787d4f4-wd6vt"]
Oct 30 01:24:35.751498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.751468    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"]
Oct 30 01:24:35.754679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.754649    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-11e7a9-6859959df5-nx7vn"]
Oct 30 01:24:35.948804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.948768    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm" podUID="57f55c7e-f518-4485-a436-fa6181614bdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:35.949223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:35.948779    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6" podUID="641d53d4-0a3c-496d-898d-bbc381ba8102" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:36.109576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:36.109541    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d9aef7-2d8f-4e94-b364-82d203e38739" path="/var/lib/kubelet/pods/00d9aef7-2d8f-4e94-b364-82d203e38739/volumes"
Oct 30 01:24:36.109961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:36.109947    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50140ed-d37d-4110-8331-71bf08574258" path="/var/lib/kubelet/pods/d50140ed-d37d-4110-8331-71bf08574258/volumes"
Oct 30 01:24:36.110323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:36.110309    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec04e918-bc46-4d4f-b2fa-735b886ea3b5" path="/var/lib/kubelet/pods/ec04e918-bc46-4d4f-b2fa-735b886ea3b5/volumes"
Oct 30 01:24:36.698548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:36.698509    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:36.992480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:36.982015    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"]
Oct 30 01:24:37.099797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:37.099752    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"]
Oct 30 01:24:37.135813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:37.135770    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b73df9b-5w2qz"]
Oct 30 01:24:37.223410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:37.223283    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:37.330931 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:37.330840    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:37.844424 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:37.844389    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"]
Oct 30 01:24:38.001032 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:38.000993    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:38.118457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:38.118295    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:38.122800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:38.122768    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c65765-23ce-437d-be07-4f136d09f263" path="/var/lib/kubelet/pods/32c65765-23ce-437d-be07-4f136d09f263/volumes"
Oct 30 01:24:38.237264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:38.237213    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:38.799535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:38.799499    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:38.917669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:38.917623    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:38.928959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:38.928928    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"]
Oct 30 01:24:38.938550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:38.938524    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4b973b-4hm4j"]
Oct 30 01:24:39.028092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:39.028058    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:39.405203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:39.405160    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:40.109411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.109370    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:24:40.113736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.113702    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d9019e-8a2b-4d5d-9ef5-e6a457805992" path="/var/lib/kubelet/pods/57d9019e-8a2b-4d5d-9ef5-e6a457805992/volumes"
Oct 30 01:24:40.163775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.163747    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:24:40.208791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.208741    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641d53d4-0a3c-496d-898d-bbc381ba8102-kube-api-access-fpgkt\" (UniqueName: \"kubernetes.io/projected/641d53d4-0a3c-496d-898d-bbc381ba8102-kube-api-access-fpgkt\") pod \"641d53d4-0a3c-496d-898d-bbc381ba8102\" (UID: \"641d53d4-0a3c-496d-898d-bbc381ba8102\") "
Oct 30 01:24:40.209005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.208814    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641d53d4-0a3c-496d-898d-bbc381ba8102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641d53d4-0a3c-496d-898d-bbc381ba8102-plugins-config-volume\") pod \"641d53d4-0a3c-496d-898d-bbc381ba8102\" (UID: \"641d53d4-0a3c-496d-898d-bbc381ba8102\") "
Oct 30 01:24:40.209356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.209317    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641d53d4-0a3c-496d-898d-bbc381ba8102-plugins-config-volume" pod "641d53d4-0a3c-496d-898d-bbc381ba8102" (UID: "641d53d4-0a3c-496d-898d-bbc381ba8102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:40.209611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.209593    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641d53d4-0a3c-496d-898d-bbc381ba8102-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:40.211834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.211810    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641d53d4-0a3c-496d-898d-bbc381ba8102-kube-api-access-fpgkt" pod "641d53d4-0a3c-496d-898d-bbc381ba8102" (UID: "641d53d4-0a3c-496d-898d-bbc381ba8102"). InnerVolumeSpecName "kube-api-access-fpgkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:40.310808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.310728    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f55c7e-f518-4485-a436-fa6181614bdf-kube-api-access-l7m6l\" (UniqueName: \"kubernetes.io/projected/57f55c7e-f518-4485-a436-fa6181614bdf-kube-api-access-l7m6l\") pod \"57f55c7e-f518-4485-a436-fa6181614bdf\" (UID: \"57f55c7e-f518-4485-a436-fa6181614bdf\") "
Oct 30 01:24:40.310971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.310858    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f55c7e-f518-4485-a436-fa6181614bdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f55c7e-f518-4485-a436-fa6181614bdf-plugins-config-volume\") pod \"57f55c7e-f518-4485-a436-fa6181614bdf\" (UID: \"57f55c7e-f518-4485-a436-fa6181614bdf\") "
Oct 30 01:24:40.311287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.311218    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f55c7e-f518-4485-a436-fa6181614bdf-plugins-config-volume" pod "57f55c7e-f518-4485-a436-fa6181614bdf" (UID: "57f55c7e-f518-4485-a436-fa6181614bdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:40.311436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.311341    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpgkt\" (UniqueName: \"kubernetes.io/projected/641d53d4-0a3c-496d-898d-bbc381ba8102-kube-api-access-fpgkt\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:40.313108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.313082    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f55c7e-f518-4485-a436-fa6181614bdf-kube-api-access-l7m6l" pod "57f55c7e-f518-4485-a436-fa6181614bdf" (UID: "57f55c7e-f518-4485-a436-fa6181614bdf"). InnerVolumeSpecName "kube-api-access-l7m6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:40.413080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.413046    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7m6l\" (UniqueName: \"kubernetes.io/projected/57f55c7e-f518-4485-a436-fa6181614bdf-kube-api-access-l7m6l\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:40.413080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.413076    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f55c7e-f518-4485-a436-fa6181614bdf-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:40.492178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.492145    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:40.667048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.666953    4576 generic.go:356] "Generic (PLEG): container finished" podID="57f55c7e-f518-4485-a436-fa6181614bdf" containerID="c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38" exitCode=0
Oct 30 01:24:40.667048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.666989    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm" event={"ID":"57f55c7e-f518-4485-a436-fa6181614bdf","Type":"ContainerDied","Data":"c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38"}
Oct 30 01:24:40.667293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.667049    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm" event={"ID":"57f55c7e-f518-4485-a436-fa6181614bdf","Type":"ContainerDied","Data":"3c60b51bf426f95956fb23997bc7a0d61f73b2174eb941e1c2cbc844515ed4ac"}
Oct 30 01:24:40.667293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.667074    4576 scope.go:122] "RemoveContainer" containerID="c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38"
Oct 30 01:24:40.667293 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.666997    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"
Oct 30 01:24:40.668902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.668878    4576 generic.go:356] "Generic (PLEG): container finished" podID="641d53d4-0a3c-496d-898d-bbc381ba8102" containerID="d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe" exitCode=0
Oct 30 01:24:40.669006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.668920    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"
Oct 30 01:24:40.669006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.668929    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6" event={"ID":"641d53d4-0a3c-496d-898d-bbc381ba8102","Type":"ContainerDied","Data":"d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe"}
Oct 30 01:24:40.669006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.668970    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6" event={"ID":"641d53d4-0a3c-496d-898d-bbc381ba8102","Type":"ContainerDied","Data":"7eb96f6c44f5b45a0a571d4d1126f1e9f11f8938c3d34e2713236d9424f669da"}
Oct 30 01:24:40.681312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.681133    4576 scope.go:122] "RemoveContainer" containerID="c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38"
Oct 30 01:24:40.681522 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:40.681499    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38\": container with ID starting with c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38 not found: ID does not exist" containerID="c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38"
Oct 30 01:24:40.681580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.681527    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38"} err="failed to get container status \"c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38\": rpc error: code = NotFound desc = could not find container \"c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38\": container with ID starting with c712c8f9a7ce5ef68bb3365217d621372169c6a54abae03ce2a71ca3d48f3b38 not found: ID does not exist"
Oct 30 01:24:40.681580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.681550    4576 scope.go:122] "RemoveContainer" containerID="d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe"
Oct 30 01:24:40.693412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.693383    4576 scope.go:122] "RemoveContainer" containerID="d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe"
Oct 30 01:24:40.693725 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:40.693704    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe\": container with ID starting with d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe not found: ID does not exist" containerID="d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe"
Oct 30 01:24:40.693782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.693733    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe"} err="failed to get container status \"d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe\": rpc error: code = NotFound desc = could not find container \"d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe\": container with ID starting with d1115e6aba0af0e02ba6e7512e7941e762f3827ed8490c93826f726dbd3906fe not found: ID does not exist"
Oct 30 01:24:40.711466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.711437    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:40.723843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.723814    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"]
Oct 30 01:24:40.742110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.740425    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-42d56b-b44c5754b-rh7c6"]
Oct 30 01:24:40.775193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.775163    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"]
Oct 30 01:24:40.778622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:40.778596    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-42d56b-67b8657855-nbzrm"]
Oct 30 01:24:41.983010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:41.982929    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l" podUID="866c83a5-1772-4d9a-af4e-1d1986d4ae14" containerName="payload-processing" containerID="cri-o://a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90" gracePeriod=25
Oct 30 01:24:42.111919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.111877    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f55c7e-f518-4485-a436-fa6181614bdf" path="/var/lib/kubelet/pods/57f55c7e-f518-4485-a436-fa6181614bdf/volumes"
Oct 30 01:24:42.113852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.113826    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641d53d4-0a3c-496d-898d-bbc381ba8102" path="/var/lib/kubelet/pods/641d53d4-0a3c-496d-898d-bbc381ba8102/volumes"
Oct 30 01:24:42.130220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.130184    4576 scope.go:122] "RemoveContainer" containerID="0939467825ef196fd208e7d7f3e1d5fa73929586438564b4f339e4a252912f7e"
Oct 30 01:24:42.142642 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.142611    4576 scope.go:122] "RemoveContainer" containerID="b230ef6d9e5d42d6b54178739780d5a8e8fb2709399ea80b3368e76bbc229fe8"
Oct 30 01:24:42.154567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.154542    4576 scope.go:122] "RemoveContainer" containerID="60be0a938d0b0b43ec9a3a5914c0139b2c30c850cf3f92166768e1d6711ce997"
Oct 30 01:24:42.166907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.166885    4576 scope.go:122] "RemoveContainer" containerID="696ccb1d5cd705629e6473626eabcf6d7f28886f84c579240349e10a06d362b4"
Oct 30 01:24:42.191378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.191348    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:42.799420 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.799382    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:24:42.799611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.799449    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:24:42.844965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.844926    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd" podUID="7a984816-b004-405c-bfb9-2bb32ed59246" containerName="payload-processing" containerID="cri-o://19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d" gracePeriod=25
Oct 30 01:24:42.979347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:42.979303    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:43.442483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:43.442449    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:44.142464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.142427    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:44.237016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.236981    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"]
Oct 30 01:24:44.250944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.250915    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-cfe8f0-bf8n4"]
Oct 30 01:24:44.252248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.252212    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:44.453846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.453815    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:44.709305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.709201    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:44.715087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.715059    4576 generic.go:356] "Generic (PLEG): container finished" podID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerID="f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29" exitCode=0
Oct 30 01:24:44.715264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.715119    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" event={"ID":"66bca5e0-3000-47b1-a675-e717c79a40a7","Type":"ContainerDied","Data":"f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29"}
Oct 30 01:24:44.715536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.715520    4576 scope.go:122] "RemoveContainer" containerID="f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29"
Oct 30 01:24:44.717185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.717156    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:24:44.717280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.717227    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:24:44.717280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:44.717274    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:24:45.608473 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.608373    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:45.725290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.725258    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" event={"ID":"66bca5e0-3000-47b1-a675-e717c79a40a7","Type":"ContainerStarted","Data":"ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e"}
Oct 30 01:24:45.737705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.737673    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:45.783034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.782153    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"]
Oct 30 01:24:45.783034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.782440    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy" containerID="cri-o://6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e" gracePeriod=30
Oct 30 01:24:45.844069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.840733    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:24:45.844069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.840887    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:24:45.844069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.840952    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 01:24:45.912484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:45.903821    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:46.093805 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.093767    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:46.107766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.107735    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ab0966-c954-43f1-8478-d31e88703b0e" path="/var/lib/kubelet/pods/92ab0966-c954-43f1-8478-d31e88703b0e/volumes"
Oct 30 01:24:46.203903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.203871    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:46.331541 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.331507    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:46.437140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437109    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"]
Oct 30 01:24:46.437668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437654    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ec04e918-bc46-4d4f-b2fa-735b886ea3b5" containerName="payload-pre-processing"
Oct 30 01:24:46.437723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437670    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec04e918-bc46-4d4f-b2fa-735b886ea3b5" containerName="payload-pre-processing"
Oct 30 01:24:46.437723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437690    4576 cpu_manager.go:414] "containerMap: removing container" podUID="641d53d4-0a3c-496d-898d-bbc381ba8102" containerName="payload-processing"
Oct 30 01:24:46.437723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437696    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641d53d4-0a3c-496d-898d-bbc381ba8102" containerName="payload-processing"
Oct 30 01:24:46.437723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437714    4576 cpu_manager.go:414] "containerMap: removing container" podUID="403dab46-3331-4dc6-a51e-a089c59b9d0d" containerName="revoke-keys"
Oct 30 01:24:46.437723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437718    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403dab46-3331-4dc6-a51e-a089c59b9d0d" containerName="revoke-keys"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437730    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437735    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437745    4576 cpu_manager.go:414] "containerMap: removing container" podUID="00d9aef7-2d8f-4e94-b364-82d203e38739" containerName="payload-processing"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437751    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d9aef7-2d8f-4e94-b364-82d203e38739" containerName="payload-processing"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437758    4576 cpu_manager.go:414] "containerMap: removing container" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437763    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437769    4576 cpu_manager.go:414] "containerMap: removing container" podUID="57f55c7e-f518-4485-a436-fa6181614bdf" containerName="payload-pre-processing"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437773    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f55c7e-f518-4485-a436-fa6181614bdf" containerName="payload-pre-processing"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437781    4576 cpu_manager.go:414] "containerMap: removing container" podUID="57d9019e-8a2b-4d5d-9ef5-e6a457805992" containerName="revoke-keys"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437785    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d9019e-8a2b-4d5d-9ef5-e6a457805992" containerName="revoke-keys"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437791    4576 cpu_manager.go:414] "containerMap: removing container" podUID="d50140ed-d37d-4110-8331-71bf08574258" containerName="payload-processing"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437795    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50140ed-d37d-4110-8331-71bf08574258" containerName="payload-processing"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437870    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="641d53d4-0a3c-496d-898d-bbc381ba8102" containerName="payload-processing"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437881    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="403dab46-3331-4dc6-a51e-a089c59b9d0d" containerName="revoke-keys"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437889    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaa2da1-a942-426f-8fb5-79d3d5be7b84" containerName="istio-proxy"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437896    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d9aef7-2d8f-4e94-b364-82d203e38739" containerName="payload-processing"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437902    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ba917c-f190-46e5-9a6a-094f37d98c6a" containerName="istio-proxy"
Oct 30 01:24:46.437898 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437908    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f55c7e-f518-4485-a436-fa6181614bdf" containerName="payload-pre-processing"
Oct 30 01:24:46.438505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437913    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d9019e-8a2b-4d5d-9ef5-e6a457805992" containerName="revoke-keys"
Oct 30 01:24:46.438505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437920    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50140ed-d37d-4110-8331-71bf08574258" containerName="payload-processing"
Oct 30 01:24:46.438505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.437926    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec04e918-bc46-4d4f-b2fa-735b886ea3b5" containerName="payload-pre-processing"
Oct 30 01:24:46.441379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.441359    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.459700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.459629    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"]
Oct 30 01:24:46.495591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.495532    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:46.498446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.498404    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:24:46.533967 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.533923    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqkd\" (UniqueName: \"kubernetes.io/projected/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-kube-api-access-mnqkd\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.534211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.534189    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.534349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.534306    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.635729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.635687    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqkd\" (UniqueName: \"kubernetes.io/projected/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-kube-api-access-mnqkd\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.636183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.635869    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.636183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.635928    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.636548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.636528    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.638606 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.638583    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.656604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.656576    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqkd\" (UniqueName: \"kubernetes.io/projected/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-kube-api-access-mnqkd\") pod \"maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.755182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.755103    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:46.841419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.841298    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:24:46.841419 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:46.841357    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 01:24:47.096876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:47.096734    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"]
Oct 30 01:24:47.097028 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:47.096996    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f43d1ba_b304_487e_a43b_ae0fdf3b0999.slice/crio-d34f16fb709fbd12d4a9faf96def7fa6d73d34545ff69fe9167e0a59f083af8b WatchSource:0}: Error finding container d34f16fb709fbd12d4a9faf96def7fa6d73d34545ff69fe9167e0a59f083af8b: Status 404 returned error can't find the container with id d34f16fb709fbd12d4a9faf96def7fa6d73d34545ff69fe9167e0a59f083af8b
Oct 30 01:24:47.741985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:47.741947    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g" event={"ID":"1f43d1ba-b304-487e-a43b-ae0fdf3b0999","Type":"ContainerStarted","Data":"8e41c1afa8f8aef49e1becf6c733f477110d47f5301ba5150d16ab2d6fd1d2d6"}
Oct 30 01:24:47.742504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:47.742032    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g" event={"ID":"1f43d1ba-b304-487e-a43b-ae0fdf3b0999","Type":"ContainerStarted","Data":"d34f16fb709fbd12d4a9faf96def7fa6d73d34545ff69fe9167e0a59f083af8b"}
Oct 30 01:24:47.800775 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:47.800665    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g" podStartSLOduration=1.800653025 podStartE2EDuration="1.800653025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:47.800266782 +0000 UTC m=+2767.871590614" watchObservedRunningTime="2026-10-30 01:24:47.800653025 +0000 UTC m=+2767.871976854"
Oct 30 01:24:47.841150 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:47.841110    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:24:47.841336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:47.841175    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 01:24:47.860731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:47.860696    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:48.405634 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:48.405594    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:24:48.750477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:48.750441    4576 generic.go:356] "Generic (PLEG): container finished" podID="1f43d1ba-b304-487e-a43b-ae0fdf3b0999" containerID="8e41c1afa8f8aef49e1becf6c733f477110d47f5301ba5150d16ab2d6fd1d2d6" exitCode=0
Oct 30 01:24:48.750874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:48.750528    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g" event={"ID":"1f43d1ba-b304-487e-a43b-ae0fdf3b0999","Type":"ContainerDied","Data":"8e41c1afa8f8aef49e1becf6c733f477110d47f5301ba5150d16ab2d6fd1d2d6"}
Oct 30 01:24:48.841537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:48.841490    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:24:48.841741 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:48.841567    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 01:24:49.841593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.841566    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:24:49.842076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.841625    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 01:24:49.948319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.948289    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:49.993914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.993841    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:49.998577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.998549    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-xks-ca\") pod \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") "
Oct 30 01:24:49.998760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.998668    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-openshift-ca\") pod \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") "
Oct 30 01:24:49.998760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.998740    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-kube-api-access-mnqkd\" (UniqueName: \"kubernetes.io/projected/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-kube-api-access-mnqkd\") pod \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\" (UID: \"1f43d1ba-b304-487e-a43b-ae0fdf3b0999\") "
Oct 30 01:24:49.999015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.998989    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-openshift-ca" pod "1f43d1ba-b304-487e-a43b-ae0fdf3b0999" (UID: "1f43d1ba-b304-487e-a43b-ae0fdf3b0999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:49.999249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:49.999218    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:50.001059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.001032    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-xks-ca" pod "1f43d1ba-b304-487e-a43b-ae0fdf3b0999" (UID: "1f43d1ba-b304-487e-a43b-ae0fdf3b0999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:50.001149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.001090    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-kube-api-access-mnqkd" pod "1f43d1ba-b304-487e-a43b-ae0fdf3b0999" (UID: "1f43d1ba-b304-487e-a43b-ae0fdf3b0999"). InnerVolumeSpecName "kube-api-access-mnqkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:50.041679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.041568    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"]
Oct 30 01:24:50.045838 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.045756    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7902333c-54zr6"]
Oct 30 01:24:50.102723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.102681    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:50.102723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.102714    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqkd\" (UniqueName: \"kubernetes.io/projected/1f43d1ba-b304-487e-a43b-ae0fdf3b0999-kube-api-access-mnqkd\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:50.111715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.111685    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514c40e0-62b3-4df2-8bf2-428d5dce63a3" path="/var/lib/kubelet/pods/514c40e0-62b3-4df2-8bf2-428d5dce63a3/volumes"
Oct 30 01:24:50.765945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.765920    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"
Oct 30 01:24:50.765945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.765932    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g" event={"ID":"1f43d1ba-b304-487e-a43b-ae0fdf3b0999","Type":"ContainerDied","Data":"d34f16fb709fbd12d4a9faf96def7fa6d73d34545ff69fe9167e0a59f083af8b"}
Oct 30 01:24:50.766162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.765974    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d34f16fb709fbd12d4a9faf96def7fa6d73d34545ff69fe9167e0a59f083af8b"
Oct 30 01:24:50.841590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.841565    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 01:24:50.841715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:50.841614    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 01:24:51.032181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.032083    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:51.111554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.111528    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:24:51.165918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.165888    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:51.178444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.178409    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l" podUID="866c83a5-1772-4d9a-af4e-1d1986d4ae14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:51.215367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215336    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-token\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-token\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.215585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215381    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-certs\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.215585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215404    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-envoy\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.215585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215456    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-data\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.215585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215491    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-socket\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.215585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215553    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f35c47-9409-4997-8a81-3d2ab56fab7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f35c47-9409-4997-8a81-3d2ab56fab7f-istiod-ca-cert\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.215858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215635    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-podinfo\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.215858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215701    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-certs" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:51.215858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215750    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-credential-socket\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.215858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.215838    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-kube-api-access-kc42s\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-kube-api-access-kc42s\") pod \"74f35c47-9409-4997-8a81-3d2ab56fab7f\" (UID: \"74f35c47-9409-4997-8a81-3d2ab56fab7f\") "
Oct 30 01:24:51.216253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.216191    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f35c47-9409-4997-8a81-3d2ab56fab7f-istiod-ca-cert" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:51.216583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.216467    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.216583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.216489    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f35c47-9409-4997-8a81-3d2ab56fab7f-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.216719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.216621    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-credential-socket" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:51.219397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.216761    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-data" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:51.219397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.217202    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-socket" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:51.219397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.218198    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-token" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:51.219397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.219119    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-kube-api-access-kc42s" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "kube-api-access-kc42s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:51.219397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.219386    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-podinfo" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:51.219724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.219593    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-envoy" pod "74f35c47-9409-4997-8a81-3d2ab56fab7f" (UID: "74f35c47-9409-4997-8a81-3d2ab56fab7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:51.307651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.307026    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"]
Oct 30 01:24:51.308186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.308156    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy" containerID="cri-o://64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48" gracePeriod=30
Oct 30 01:24:51.317987 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.317955    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"]
Oct 30 01:24:51.318830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.318803    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.318830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.318832    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.318990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.318846    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc42s\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-kube-api-access-kc42s\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.318990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.318859    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.318990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.318872    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.318990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.318883    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.318990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.318896    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f35c47-9409-4997-8a81-3d2ab56fab7f-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:51.362727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.362685    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:51.589581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.589501    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:51.773870 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.773838    4576 generic.go:356] "Generic (PLEG): container finished" podID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerID="6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e" exitCode=0
Oct 30 01:24:51.774076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.773913    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"
Oct 30 01:24:51.774076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.773911    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" event={"ID":"74f35c47-9409-4997-8a81-3d2ab56fab7f","Type":"ContainerDied","Data":"6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e"}
Oct 30 01:24:51.774076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.774041    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st" event={"ID":"74f35c47-9409-4997-8a81-3d2ab56fab7f","Type":"ContainerDied","Data":"8782c9a60b353bdc4fb8b9f282d83985b1933b6ee1bd5a110a98afbce6d9302b"}
Oct 30 01:24:51.774076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.774073    4576 scope.go:122] "RemoveContainer" containerID="6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e"
Oct 30 01:24:51.785997 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.785975    4576 scope.go:122] "RemoveContainer" containerID="6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e"
Oct 30 01:24:51.786330 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:51.786310    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e\": container with ID starting with 6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e not found: ID does not exist" containerID="6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e"
Oct 30 01:24:51.786404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.786342    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e"} err="failed to get container status \"6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e\": rpc error: code = NotFound desc = could not find container \"6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e\": container with ID starting with 6124d47909908a19a309dff1881ad454414f59717752e6a03f904ed1b9a2cd1e not found: ID does not exist"
Oct 30 01:24:51.794792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.794762    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:51.831856 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.831828    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"]
Oct 30 01:24:51.856253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.853841    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b973b-openshift-default-6476bc68df-bf4st"]
Oct 30 01:24:51.856253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.854130    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:24:51.856253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.855567    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:24:51.862769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.857882    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:24:51.919162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.919130    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:51.970073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.970023    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"]
Oct 30 01:24:51.970916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.970892    4576 cpu_manager.go:414] "containerMap: removing container" podUID="1f43d1ba-b304-487e-a43b-ae0fdf3b0999" containerName="revoke-keys"
Oct 30 01:24:51.970972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.970920    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f43d1ba-b304-487e-a43b-ae0fdf3b0999" containerName="revoke-keys"
Oct 30 01:24:51.970972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.970939    4576 cpu_manager.go:414] "containerMap: removing container" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy"
Oct 30 01:24:51.970972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.970948    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy"
Oct 30 01:24:51.971121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.971108    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f43d1ba-b304-487e-a43b-ae0fdf3b0999" containerName="revoke-keys"
Oct 30 01:24:51.971172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.971126    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" containerName="istio-proxy"
Oct 30 01:24:51.976464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.976438    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:51.991712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:51.991682    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"]
Oct 30 01:24:52.014372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.014343    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:52.014598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.014564    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:24:52.028019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.027989    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5r9\" (UniqueName: \"kubernetes.io/projected/e14266b3-4133-4451-8c50-6f86ddf04574-kube-api-access-jj5r9\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.028303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.028057    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.028303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.028117    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.109905 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.109831    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f35c47-9409-4997-8a81-3d2ab56fab7f" path="/var/lib/kubelet/pods/74f35c47-9409-4997-8a81-3d2ab56fab7f/volumes"
Oct 30 01:24:52.129469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.129437    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.129652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.129555    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5r9\" (UniqueName: \"kubernetes.io/projected/e14266b3-4133-4451-8c50-6f86ddf04574-kube-api-access-jj5r9\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.129652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.129598    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.130125 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.130099    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.132099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.132080    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.150436 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.150411    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5r9\" (UniqueName: \"kubernetes.io/projected/e14266b3-4133-4451-8c50-6f86ddf04574-kube-api-access-jj5r9\") pod \"maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.301015 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.300981    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:52.604050 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.604024    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"]
Oct 30 01:24:52.606330 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:52.606295    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14266b3_4133_4451_8c50_6f86ddf04574.slice/crio-005551002b5e706f0b4a5c6f79bf38338563c1ae1a8a4375d469cfb8af2c420c WatchSource:0}: Error finding container 005551002b5e706f0b4a5c6f79bf38338563c1ae1a8a4375d469cfb8af2c420c: Status 404 returned error can't find the container with id 005551002b5e706f0b4a5c6f79bf38338563c1ae1a8a4375d469cfb8af2c420c
Oct 30 01:24:52.746208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.746169    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd" podUID="7a984816-b004-405c-bfb9-2bb32ed59246" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:52.784730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:52.784605    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt" event={"ID":"e14266b3-4133-4451-8c50-6f86ddf04574","Type":"ContainerStarted","Data":"005551002b5e706f0b4a5c6f79bf38338563c1ae1a8a4375d469cfb8af2c420c"}
Oct 30 01:24:53.115715 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:53.115678    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:53.271223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:53.271180    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:53.794062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:53.793973    4576 generic.go:356] "Generic (PLEG): container finished" podID="e14266b3-4133-4451-8c50-6f86ddf04574" containerID="425a124368d5221c237d376f2f7ac4e0a3b4e7def1765081edb00f59f3b040e8" exitCode=0
Oct 30 01:24:53.794265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:53.794057    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt" event={"ID":"e14266b3-4133-4451-8c50-6f86ddf04574","Type":"ContainerDied","Data":"425a124368d5221c237d376f2f7ac4e0a3b4e7def1765081edb00f59f3b040e8"}
Oct 30 01:24:54.230114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:54.230082    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:54.786043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:54.786009    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:54.927935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:54.927900    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:54.995769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:54.995741    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:55.042646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.042551    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:55.058204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.058151    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"]
Oct 30 01:24:55.062315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.062285    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-d0ae0d-txtg8"]
Oct 30 01:24:55.090488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.090459    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14266b3-4133-4451-8c50-6f86ddf04574-kube-api-access-jj5r9\" (UniqueName: \"kubernetes.io/projected/e14266b3-4133-4451-8c50-6f86ddf04574-kube-api-access-jj5r9\") pod \"e14266b3-4133-4451-8c50-6f86ddf04574\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") "
Oct 30 01:24:55.090659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.090495    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-xks-ca\") pod \"e14266b3-4133-4451-8c50-6f86ddf04574\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") "
Oct 30 01:24:55.090731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.090703    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-openshift-ca\") pod \"e14266b3-4133-4451-8c50-6f86ddf04574\" (UID: \"e14266b3-4133-4451-8c50-6f86ddf04574\") "
Oct 30 01:24:55.091041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.091015    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-openshift-ca" pod "e14266b3-4133-4451-8c50-6f86ddf04574" (UID: "e14266b3-4133-4451-8c50-6f86ddf04574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:55.091362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.091342    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:55.093350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.093328    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-xks-ca" pod "e14266b3-4133-4451-8c50-6f86ddf04574" (UID: "e14266b3-4133-4451-8c50-6f86ddf04574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:55.093459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.093374    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14266b3-4133-4451-8c50-6f86ddf04574-kube-api-access-jj5r9" pod "e14266b3-4133-4451-8c50-6f86ddf04574" (UID: "e14266b3-4133-4451-8c50-6f86ddf04574"). InnerVolumeSpecName "kube-api-access-jj5r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:55.194493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.193886    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj5r9\" (UniqueName: \"kubernetes.io/projected/e14266b3-4133-4451-8c50-6f86ddf04574-kube-api-access-jj5r9\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:55.194493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.193918    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14266b3-4133-4451-8c50-6f86ddf04574-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:55.484978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.484945    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:55.595878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.595840    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:55.734470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.734434    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"]
Oct 30 01:24:55.748395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.748311    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-bdf6f3-kjpb8"]
Oct 30 01:24:55.810771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.810740    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt" event={"ID":"e14266b3-4133-4451-8c50-6f86ddf04574","Type":"ContainerDied","Data":"005551002b5e706f0b4a5c6f79bf38338563c1ae1a8a4375d469cfb8af2c420c"}
Oct 30 01:24:55.810771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.810774    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"
Oct 30 01:24:55.810982 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.810787    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="005551002b5e706f0b4a5c6f79bf38338563c1ae1a8a4375d469cfb8af2c420c"
Oct 30 01:24:55.823046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:55.823017    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:56.074350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.074263    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:56.108577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.108547    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403dab46-3331-4dc6-a51e-a089c59b9d0d" path="/var/lib/kubelet/pods/403dab46-3331-4dc6-a51e-a089c59b9d0d/volumes"
Oct 30 01:24:56.108990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.108976    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446406f2-5194-45db-bd67-da815db637b5" path="/var/lib/kubelet/pods/446406f2-5194-45db-bd67-da815db637b5/volumes"
Oct 30 01:24:56.597086 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.597031    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.710584    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-envoy\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.710648    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-credential-socket\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.710719    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-kube-api-access-tt9pk\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-kube-api-access-tt9pk\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.710765    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083c326c-dc0c-4720-b2f4-7e100846e52a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c326c-dc0c-4720-b2f4-7e100846e52a-istiod-ca-cert\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.710819    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-token\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-token\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.710960    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-data\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.711024    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-podinfo\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.711060    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-socket\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.711119    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-certs\") pod \"083c326c-dc0c-4720-b2f4-7e100846e52a\" (UID: \"083c326c-dc0c-4720-b2f4-7e100846e52a\") "
Oct 30 01:24:56.711867 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.711818    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-certs" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:56.713536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.713339    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-credential-socket" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:56.713536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.713506    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083c326c-dc0c-4720-b2f4-7e100846e52a-istiod-ca-cert" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:56.715780 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.715745    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-socket" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:56.720833 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.720796    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-data" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:56.736118 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.736077    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-kube-api-access-tt9pk" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "kube-api-access-tt9pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:56.737940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.737859    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-token" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:56.739685 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.739560    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-podinfo" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:56.741064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.741031    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-envoy" pod "083c326c-dc0c-4720-b2f4-7e100846e52a" (UID: "083c326c-dc0c-4720-b2f4-7e100846e52a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:56.777099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.777070    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"]
Oct 30 01:24:56.780941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.780912    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"]
Oct 30 01:24:56.813329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813298    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.813329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813327    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.813329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813337    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.813595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813345    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.813595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813353    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.813595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813361    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c326c-dc0c-4720-b2f4-7e100846e52a-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.813595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813370    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt9pk\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-kube-api-access-tt9pk\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.813595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813378    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c326c-dc0c-4720-b2f4-7e100846e52a-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.813595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.813386    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c326c-dc0c-4720-b2f4-7e100846e52a-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:24:56.819048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.819017    4576 generic.go:356] "Generic (PLEG): container finished" podID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerID="64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48" exitCode=0
Oct 30 01:24:56.819482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.819087    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" event={"ID":"083c326c-dc0c-4720-b2f4-7e100846e52a","Type":"ContainerDied","Data":"64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48"}
Oct 30 01:24:56.819482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.819168    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw" event={"ID":"083c326c-dc0c-4720-b2f4-7e100846e52a","Type":"ContainerDied","Data":"4dcc1a77e98036c3f6dbec3397f024b7e4f5cad03d0c10c6efdeb65d517eb2e2"}
Oct 30 01:24:56.819482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.819215    4576 scope.go:122] "RemoveContainer" containerID="64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48"
Oct 30 01:24:56.819482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.819322    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"
Oct 30 01:24:56.833137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.833117    4576 scope.go:122] "RemoveContainer" containerID="64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48"
Oct 30 01:24:56.833526 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:56.833504    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48\": container with ID starting with 64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48 not found: ID does not exist" containerID="64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48"
Oct 30 01:24:56.833608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.833541    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48"} err="failed to get container status \"64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48\": rpc error: code = NotFound desc = could not find container \"64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48\": container with ID starting with 64f104a4d19f86a862b25be4b04679b6a055af768c8d1a4cb8c2413ac348bb48 not found: ID does not exist"
Oct 30 01:24:56.905817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.905731    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:56.913860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.913829    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"]
Oct 30 01:24:56.930884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.930850    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"]
Oct 30 01:24:56.940792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.940766    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cfe8f0-openshift-default-7946898fdf-79pjw"]
Oct 30 01:24:56.953137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:56.953114    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"]
Oct 30 01:24:57.081071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.081040    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:57.189205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.189156    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:57.368273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.368226    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"]
Oct 30 01:24:57.368857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.368837    4576 cpu_manager.go:414] "containerMap: removing container" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy"
Oct 30 01:24:57.368857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.368858    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy"
Oct 30 01:24:57.369065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.368892    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e14266b3-4133-4451-8c50-6f86ddf04574" containerName="revoke-keys"
Oct 30 01:24:57.369065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.368900    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14266b3-4133-4451-8c50-6f86ddf04574" containerName="revoke-keys"
Oct 30 01:24:57.369065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.369031    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" containerName="istio-proxy"
Oct 30 01:24:57.369065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.369050    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14266b3-4133-4451-8c50-6f86ddf04574" containerName="revoke-keys"
Oct 30 01:24:57.373913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.373890    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.389154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.389128    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"]
Oct 30 01:24:57.391159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.391116    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:24:57.399397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.399377    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:24:57.401221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.401203    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"]
Oct 30 01:24:57.401347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.401309    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.417221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.417192    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"]
Oct 30 01:24:57.521782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.521704    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.521782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.521761    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.522001 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.521916    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bvt\" (UniqueName: \"kubernetes.io/projected/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-kube-api-access-f7bvt\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.522001 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.521963    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.522001 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.521983    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.522167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.522092    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p7cj\" (UniqueName: \"kubernetes.io/projected/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-kube-api-access-6p7cj\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.630160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.630120    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.630716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.630276    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bvt\" (UniqueName: \"kubernetes.io/projected/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-kube-api-access-f7bvt\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.630716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.630333    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.630716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.630362    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.630716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.630398    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p7cj\" (UniqueName: \"kubernetes.io/projected/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-kube-api-access-6p7cj\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.630716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.630503    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.631918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.631884    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.632523 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.632499    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.645926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.645897    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.651088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.651058    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.655306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.654260    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bvt\" (UniqueName: \"kubernetes.io/projected/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-kube-api-access-f7bvt\") pod \"maas-api-revoke-sub-e7d25ff4-24tc4\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.655306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.654627    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p7cj\" (UniqueName: \"kubernetes.io/projected/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-kube-api-access-6p7cj\") pod \"maas-api-revoke-sub-d036e807-llzwl\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.687491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.687450    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:24:57.718515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.718490    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:24:57.785556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.785474    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:57.992020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:57.991992    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"]
Oct 30 01:24:57.993269 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:57.993211    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153e78da_54d6_4f12_8aff_d7d0bb9e57cd.slice/crio-763aefa493b517d4fda365636adfa47f74c4b2d304ed261003c4c25961f900ab WatchSource:0}: Error finding container 763aefa493b517d4fda365636adfa47f74c4b2d304ed261003c4c25961f900ab: Status 404 returned error can't find the container with id 763aefa493b517d4fda365636adfa47f74c4b2d304ed261003c4c25961f900ab
Oct 30 01:24:58.045307 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:24:58.045269    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffc843b_dc8e_4e6b_a64e_46b11a2e51e7.slice/crio-ade26c289a30873afada887063e3169435ccda0c950e06fa83d48deb6d5c5606 WatchSource:0}: Error finding container ade26c289a30873afada887063e3169435ccda0c950e06fa83d48deb6d5c5606: Status 404 returned error can't find the container with id ade26c289a30873afada887063e3169435ccda0c950e06fa83d48deb6d5c5606
Oct 30 01:24:58.046071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:58.046038    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"]
Oct 30 01:24:58.109153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:58.109123    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083c326c-dc0c-4720-b2f4-7e100846e52a" path="/var/lib/kubelet/pods/083c326c-dc0c-4720-b2f4-7e100846e52a/volumes"
Oct 30 01:24:58.844992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:58.844940    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl" event={"ID":"153e78da-54d6-4f12-8aff-d7d0bb9e57cd","Type":"ContainerStarted","Data":"f4c8429da209c48830a428e6a30342754d693c7f4fd524255cfa6f8920821f49"}
Oct 30 01:24:58.845469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:58.845009    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl" event={"ID":"153e78da-54d6-4f12-8aff-d7d0bb9e57cd","Type":"ContainerStarted","Data":"763aefa493b517d4fda365636adfa47f74c4b2d304ed261003c4c25961f900ab"}
Oct 30 01:24:58.846940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:58.846908    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4" event={"ID":"effc843b-dc8e-4e6b-a64e-46b11a2e51e7","Type":"ContainerStarted","Data":"12bc850d4e946405642a7c9d8fd4468a92930e3241464de3883bb7c34804b2da"}
Oct 30 01:24:58.847073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:58.846965    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4" event={"ID":"effc843b-dc8e-4e6b-a64e-46b11a2e51e7","Type":"ContainerStarted","Data":"ade26c289a30873afada887063e3169435ccda0c950e06fa83d48deb6d5c5606"}
Oct 30 01:24:58.887202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:58.887155    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl" podStartSLOduration=1.88714312 podStartE2EDuration="1.88714312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:58.88589038 +0000 UTC m=+2778.957214228" watchObservedRunningTime="2026-10-30 01:24:58.88714312 +0000 UTC m=+2778.958466955"
Oct 30 01:24:58.928697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:58.928640    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4" podStartSLOduration=1.928628171 podStartE2EDuration="1.928628171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:58.927485008 +0000 UTC m=+2778.998808845" watchObservedRunningTime="2026-10-30 01:24:58.928628171 +0000 UTC m=+2778.999952005"
Oct 30 01:24:59.028204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:59.028163    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:24:59.041616 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:24:59.041569    4576 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffc843b_dc8e_4e6b_a64e_46b11a2e51e7.slice/crio-conmon-12bc850d4e946405642a7c9d8fd4468a92930e3241464de3883bb7c34804b2da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffc843b_dc8e_4e6b_a64e_46b11a2e51e7.slice/crio-12bc850d4e946405642a7c9d8fd4468a92930e3241464de3883bb7c34804b2da.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 01:24:59.318585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:59.318546    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:24:59.855943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:59.855904    4576 generic.go:356] "Generic (PLEG): container finished" podID="153e78da-54d6-4f12-8aff-d7d0bb9e57cd" containerID="f4c8429da209c48830a428e6a30342754d693c7f4fd524255cfa6f8920821f49" exitCode=0
Oct 30 01:24:59.856377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:59.855984    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl" event={"ID":"153e78da-54d6-4f12-8aff-d7d0bb9e57cd","Type":"ContainerDied","Data":"f4c8429da209c48830a428e6a30342754d693c7f4fd524255cfa6f8920821f49"}
Oct 30 01:24:59.857635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:59.857612    4576 generic.go:356] "Generic (PLEG): container finished" podID="effc843b-dc8e-4e6b-a64e-46b11a2e51e7" containerID="12bc850d4e946405642a7c9d8fd4468a92930e3241464de3883bb7c34804b2da" exitCode=0
Oct 30 01:24:59.857752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:24:59.857669    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4" event={"ID":"effc843b-dc8e-4e6b-a64e-46b11a2e51e7","Type":"ContainerDied","Data":"12bc850d4e946405642a7c9d8fd4468a92930e3241464de3883bb7c34804b2da"}
Oct 30 01:25:01.135819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.135784    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:25:01.175369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.175333    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l" podUID="866c83a5-1772-4d9a-af4e-1d1986d4ae14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:25:01.192687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.192645    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:25:01.216915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.216877    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-kube-api-access-6p7cj\" (UniqueName: \"kubernetes.io/projected/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-kube-api-access-6p7cj\") pod \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") "
Oct 30 01:25:01.217099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.217028    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-xks-ca\") pod \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") "
Oct 30 01:25:01.217099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.217073    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-xks-ca\") pod \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") "
Oct 30 01:25:01.217310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.217106    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-kube-api-access-f7bvt\" (UniqueName: \"kubernetes.io/projected/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-kube-api-access-f7bvt\") pod \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") "
Oct 30 01:25:01.217575 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.217550    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-openshift-ca\") pod \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\" (UID: \"effc843b-dc8e-4e6b-a64e-46b11a2e51e7\") "
Oct 30 01:25:01.217918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.217894    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-openshift-ca" pod "effc843b-dc8e-4e6b-a64e-46b11a2e51e7" (UID: "effc843b-dc8e-4e6b-a64e-46b11a2e51e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:01.218114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.218095    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-openshift-ca\") pod \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\" (UID: \"153e78da-54d6-4f12-8aff-d7d0bb9e57cd\") "
Oct 30 01:25:01.218477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.218455    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-openshift-ca" pod "153e78da-54d6-4f12-8aff-d7d0bb9e57cd" (UID: "153e78da-54d6-4f12-8aff-d7d0bb9e57cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:01.219006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.218985    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:01.219107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.219008    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:01.220262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.220205    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-kube-api-access-f7bvt" pod "effc843b-dc8e-4e6b-a64e-46b11a2e51e7" (UID: "effc843b-dc8e-4e6b-a64e-46b11a2e51e7"). InnerVolumeSpecName "kube-api-access-f7bvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:01.220587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.220556    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-kube-api-access-6p7cj" pod "153e78da-54d6-4f12-8aff-d7d0bb9e57cd" (UID: "153e78da-54d6-4f12-8aff-d7d0bb9e57cd"). InnerVolumeSpecName "kube-api-access-6p7cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:01.220790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.220765    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-xks-ca" pod "153e78da-54d6-4f12-8aff-d7d0bb9e57cd" (UID: "153e78da-54d6-4f12-8aff-d7d0bb9e57cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:01.221589 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.221566    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-xks-ca" pod "effc843b-dc8e-4e6b-a64e-46b11a2e51e7" (UID: "effc843b-dc8e-4e6b-a64e-46b11a2e51e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:01.320244 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.320200    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:01.320244 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.320250    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:01.320447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.320260    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bvt\" (UniqueName: \"kubernetes.io/projected/effc843b-dc8e-4e6b-a64e-46b11a2e51e7-kube-api-access-f7bvt\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:01.320447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.320271    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p7cj\" (UniqueName: \"kubernetes.io/projected/153e78da-54d6-4f12-8aff-d7d0bb9e57cd-kube-api-access-6p7cj\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:01.411772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.411679    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:01.875646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.875614    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"
Oct 30 01:25:01.875819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.875717    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl" event={"ID":"153e78da-54d6-4f12-8aff-d7d0bb9e57cd","Type":"ContainerDied","Data":"763aefa493b517d4fda365636adfa47f74c4b2d304ed261003c4c25961f900ab"}
Oct 30 01:25:01.875819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.875770    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="763aefa493b517d4fda365636adfa47f74c4b2d304ed261003c4c25961f900ab"
Oct 30 01:25:01.877480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.877458    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"
Oct 30 01:25:01.877645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.877518    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4" event={"ID":"effc843b-dc8e-4e6b-a64e-46b11a2e51e7","Type":"ContainerDied","Data":"ade26c289a30873afada887063e3169435ccda0c950e06fa83d48deb6d5c5606"}
Oct 30 01:25:01.877645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.877561    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ade26c289a30873afada887063e3169435ccda0c950e06fa83d48deb6d5c5606"
Oct 30 01:25:01.914716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.914667    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst" podUID="7b1bc42f-9481-44bd-91a0-53f9454889c5" containerName="payload-processing" containerID="cri-o://e7fdb30aaf9d394afbb58facfe2c240789f51d180f3784ad8b9389bf0f84ba18" gracePeriod=25
Oct 30 01:25:01.954326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:01.954285    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n" podUID="cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" containerName="payload-pre-processing" containerID="cri-o://9adb3e81f02a1516725e97bd04cea725887c5d34ea367a52ae677d136b8aa4fc" gracePeriod=25
Oct 30 01:25:02.233281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.233220    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:02.247407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.247387    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:25:02.334004 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.333959    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/866c83a5-1772-4d9a-af4e-1d1986d4ae14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866c83a5-1772-4d9a-af4e-1d1986d4ae14-plugins-config-volume\") pod \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\" (UID: \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\") "
Oct 30 01:25:02.334211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.334064    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866c83a5-1772-4d9a-af4e-1d1986d4ae14-kube-api-access-l452x\" (UniqueName: \"kubernetes.io/projected/866c83a5-1772-4d9a-af4e-1d1986d4ae14-kube-api-access-l452x\") pod \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\" (UID: \"866c83a5-1772-4d9a-af4e-1d1986d4ae14\") "
Oct 30 01:25:02.334373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.334344    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866c83a5-1772-4d9a-af4e-1d1986d4ae14-plugins-config-volume" pod "866c83a5-1772-4d9a-af4e-1d1986d4ae14" (UID: "866c83a5-1772-4d9a-af4e-1d1986d4ae14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:02.334726 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.334701    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866c83a5-1772-4d9a-af4e-1d1986d4ae14-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:02.336496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.336467    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866c83a5-1772-4d9a-af4e-1d1986d4ae14-kube-api-access-l452x" pod "866c83a5-1772-4d9a-af4e-1d1986d4ae14" (UID: "866c83a5-1772-4d9a-af4e-1d1986d4ae14"). InnerVolumeSpecName "kube-api-access-l452x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:02.363429 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.363395    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:02.436149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.436062    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l452x\" (UniqueName: \"kubernetes.io/projected/866c83a5-1772-4d9a-af4e-1d1986d4ae14-kube-api-access-l452x\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:02.524926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.518042    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"]
Oct 30 01:25:02.524926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.518302    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy" containerID="cri-o://6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a" gracePeriod=30
Oct 30 01:25:02.571971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.571931    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:02.746660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.746614    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd" podUID="7a984816-b004-405c-bfb9-2bb32ed59246" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:25:02.838400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.838365    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:02.887544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.887515    4576 generic.go:356] "Generic (PLEG): container finished" podID="866c83a5-1772-4d9a-af4e-1d1986d4ae14" containerID="a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90" exitCode=0
Oct 30 01:25:02.887745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.887567    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"
Oct 30 01:25:02.887745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.887597    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l" event={"ID":"866c83a5-1772-4d9a-af4e-1d1986d4ae14","Type":"ContainerDied","Data":"a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90"}
Oct 30 01:25:02.887745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.887665    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l" event={"ID":"866c83a5-1772-4d9a-af4e-1d1986d4ae14","Type":"ContainerDied","Data":"94c99f9c145cd123983cd6c787286c3f4a14d9c514e3616aa5fe5774fa98eb0f"}
Oct 30 01:25:02.887745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.887696    4576 scope.go:122] "RemoveContainer" containerID="a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90"
Oct 30 01:25:02.947468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.947435    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"]
Oct 30 01:25:02.952625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.952589    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:02.954437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.954413    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b973b-85748d584b-xnh5l"]
Oct 30 01:25:02.958918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.958774    4576 scope.go:122] "RemoveContainer" containerID="a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90"
Oct 30 01:25:02.959130 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:02.959110    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90\": container with ID starting with a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90 not found: ID does not exist" containerID="a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90"
Oct 30 01:25:02.959194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:02.959143    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90"} err="failed to get container status \"a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90\": rpc error: code = NotFound desc = could not find container \"a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90\": container with ID starting with a790a6dc1078e0ad06dbd4c4d993bb73bbaaf5320c5e89963c3fed68f3935f90 not found: ID does not exist"
Oct 30 01:25:03.082637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.079463    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:03.140847 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.140402    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:25:03.150673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.150646    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a984816-b004-405c-bfb9-2bb32ed59246-kube-api-access-n6nsz\" (UniqueName: \"kubernetes.io/projected/7a984816-b004-405c-bfb9-2bb32ed59246-kube-api-access-n6nsz\") pod \"7a984816-b004-405c-bfb9-2bb32ed59246\" (UID: \"7a984816-b004-405c-bfb9-2bb32ed59246\") "
Oct 30 01:25:03.150854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.150737    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a984816-b004-405c-bfb9-2bb32ed59246-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a984816-b004-405c-bfb9-2bb32ed59246-plugins-config-volume\") pod \"7a984816-b004-405c-bfb9-2bb32ed59246\" (UID: \"7a984816-b004-405c-bfb9-2bb32ed59246\") "
Oct 30 01:25:03.151360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.151167    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a984816-b004-405c-bfb9-2bb32ed59246-plugins-config-volume" pod "7a984816-b004-405c-bfb9-2bb32ed59246" (UID: "7a984816-b004-405c-bfb9-2bb32ed59246"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:03.153020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.152997    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a984816-b004-405c-bfb9-2bb32ed59246-kube-api-access-n6nsz" pod "7a984816-b004-405c-bfb9-2bb32ed59246" (UID: "7a984816-b004-405c-bfb9-2bb32ed59246"). InnerVolumeSpecName "kube-api-access-n6nsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:03.169126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169099    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"]
Oct 30 01:25:03.169652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169636    4576 cpu_manager.go:414] "containerMap: removing container" podUID="7a984816-b004-405c-bfb9-2bb32ed59246" containerName="payload-processing"
Oct 30 01:25:03.169733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169653    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a984816-b004-405c-bfb9-2bb32ed59246" containerName="payload-processing"
Oct 30 01:25:03.169733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169666    4576 cpu_manager.go:414] "containerMap: removing container" podUID="866c83a5-1772-4d9a-af4e-1d1986d4ae14" containerName="payload-processing"
Oct 30 01:25:03.169733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169671    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="866c83a5-1772-4d9a-af4e-1d1986d4ae14" containerName="payload-processing"
Oct 30 01:25:03.169733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169694    4576 cpu_manager.go:414] "containerMap: removing container" podUID="153e78da-54d6-4f12-8aff-d7d0bb9e57cd" containerName="revoke-keys"
Oct 30 01:25:03.169733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169699    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153e78da-54d6-4f12-8aff-d7d0bb9e57cd" containerName="revoke-keys"
Oct 30 01:25:03.169733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169716    4576 cpu_manager.go:414] "containerMap: removing container" podUID="effc843b-dc8e-4e6b-a64e-46b11a2e51e7" containerName="revoke-keys"
Oct 30 01:25:03.169733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169722    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effc843b-dc8e-4e6b-a64e-46b11a2e51e7" containerName="revoke-keys"
Oct 30 01:25:03.169999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169799    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="effc843b-dc8e-4e6b-a64e-46b11a2e51e7" containerName="revoke-keys"
Oct 30 01:25:03.169999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169810    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a984816-b004-405c-bfb9-2bb32ed59246" containerName="payload-processing"
Oct 30 01:25:03.169999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169815    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="866c83a5-1772-4d9a-af4e-1d1986d4ae14" containerName="payload-processing"
Oct 30 01:25:03.169999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.169824    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="153e78da-54d6-4f12-8aff-d7d0bb9e57cd" containerName="revoke-keys"
Oct 30 01:25:03.173220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.173201    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.190116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.190091    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:03.190368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.190338    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:25:03.195404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.195373    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"]
Oct 30 01:25:03.252580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.252539    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4hz\" (UniqueName: \"kubernetes.io/projected/621dfe32-294e-4eb1-9afb-919e415182d3-kube-api-access-tp4hz\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.253042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.252676    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.253042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.252794    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.253042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.252893    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6nsz\" (UniqueName: \"kubernetes.io/projected/7a984816-b004-405c-bfb9-2bb32ed59246-kube-api-access-n6nsz\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:03.253042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.252910    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a984816-b004-405c-bfb9-2bb32ed59246-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:03.274519 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.274438    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:03.355167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.355137    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.355400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.355190    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.355400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.355338    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4hz\" (UniqueName: \"kubernetes.io/projected/621dfe32-294e-4eb1-9afb-919e415182d3-kube-api-access-tp4hz\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.355975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.355947    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.358916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.358887    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.379711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.379670    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4hz\" (UniqueName: \"kubernetes.io/projected/621dfe32-294e-4eb1-9afb-919e415182d3-kube-api-access-tp4hz\") pod \"maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.488053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.488020    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:03.738941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.738899    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:03.791860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.791809    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"]
Oct 30 01:25:03.793151 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:25:03.793122    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621dfe32_294e_4eb1_9afb_919e415182d3.slice/crio-0c256a676cd6780cd3718c2f89caa2d10f7f779689e2aaf11ed049314e80b20b WatchSource:0}: Error finding container 0c256a676cd6780cd3718c2f89caa2d10f7f779689e2aaf11ed049314e80b20b: Status 404 returned error can't find the container with id 0c256a676cd6780cd3718c2f89caa2d10f7f779689e2aaf11ed049314e80b20b
Oct 30 01:25:03.859057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.859024    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:03.888100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.888061    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"]
Oct 30 01:25:03.894649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.894608    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:03.901025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.900995    4576 generic.go:356] "Generic (PLEG): container finished" podID="7a984816-b004-405c-bfb9-2bb32ed59246" containerID="19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d" exitCode=0
Oct 30 01:25:03.901187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.901040    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"
Oct 30 01:25:03.901187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.901072    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd" event={"ID":"7a984816-b004-405c-bfb9-2bb32ed59246","Type":"ContainerDied","Data":"19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d"}
Oct 30 01:25:03.901187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.901122    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd" event={"ID":"7a984816-b004-405c-bfb9-2bb32ed59246","Type":"ContainerDied","Data":"9a7fcfae9c12f6ea3099199f11de18e6897b882c1daa593dcd44f3ad9c3346d8"}
Oct 30 01:25:03.901187 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.901151    4576 scope.go:122] "RemoveContainer" containerID="19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d"
Oct 30 01:25:03.902945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.902862    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp" event={"ID":"621dfe32-294e-4eb1-9afb-919e415182d3","Type":"ContainerStarted","Data":"0c256a676cd6780cd3718c2f89caa2d10f7f779689e2aaf11ed049314e80b20b"}
Oct 30 01:25:03.908974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.908950    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"]
Oct 30 01:25:03.918326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.918302    4576 scope.go:122] "RemoveContainer" containerID="19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d"
Oct 30 01:25:03.918648 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:03.918621    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d\": container with ID starting with 19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d not found: ID does not exist" containerID="19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d"
Oct 30 01:25:03.918751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.918661    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d"} err="failed to get container status \"19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d\": rpc error: code = NotFound desc = could not find container \"19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d\": container with ID starting with 19bce22f4bd401b63d229fc64221cbffbf986b6d4414028ce5e837e617472a6d not found: ID does not exist"
Oct 30 01:25:03.964116 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.964085    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:03.964283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.964122    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qp7\" (UniqueName: \"kubernetes.io/projected/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-kube-api-access-c7qp7\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:03.964348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.964298    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:03.990047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:03.990014    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"]
Oct 30 01:25:04.002923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.002897    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-d0ae0d-7886d899d6-zrmxd"]
Oct 30 01:25:04.065193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.065113    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:04.065193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.065154    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qp7\" (UniqueName: \"kubernetes.io/projected/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-kube-api-access-c7qp7\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:04.065414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.065216    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:04.065863 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.065842    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:04.067801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.067779    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:04.098358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.098332    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qp7\" (UniqueName: \"kubernetes.io/projected/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-kube-api-access-c7qp7\") pod \"maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:04.109720 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.109690    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a984816-b004-405c-bfb9-2bb32ed59246" path="/var/lib/kubelet/pods/7a984816-b004-405c-bfb9-2bb32ed59246/volumes"
Oct 30 01:25:04.110114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.110099    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="866c83a5-1772-4d9a-af4e-1d1986d4ae14" path="/var/lib/kubelet/pods/866c83a5-1772-4d9a-af4e-1d1986d4ae14/volumes"
Oct 30 01:25:04.178352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.178321    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:04.222539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.222500    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:04.532412 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.532390    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"]
Oct 30 01:25:04.534939 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:25:04.534911    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f04712_8b58_4e3d_9ac0_f415dbe3f7ce.slice/crio-0fc793365b0c48daf0bcdd5828f1cb701a918b3fcfeee294e39aa1d54fb2dddd WatchSource:0}: Error finding container 0fc793365b0c48daf0bcdd5828f1cb701a918b3fcfeee294e39aa1d54fb2dddd: Status 404 returned error can't find the container with id 0fc793365b0c48daf0bcdd5828f1cb701a918b3fcfeee294e39aa1d54fb2dddd
Oct 30 01:25:04.652358 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.652321    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:04.783302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.783177    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:04.913614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.913583    4576 generic.go:356] "Generic (PLEG): container finished" podID="621dfe32-294e-4eb1-9afb-919e415182d3" containerID="da32b6401dacbb6a0ea1033601436dd47f3e2e2a77476376a1b6fb4d8ca2a99e" exitCode=0
Oct 30 01:25:04.913809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.913663    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp" event={"ID":"621dfe32-294e-4eb1-9afb-919e415182d3","Type":"ContainerDied","Data":"da32b6401dacbb6a0ea1033601436dd47f3e2e2a77476376a1b6fb4d8ca2a99e"}
Oct 30 01:25:04.915032 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:04.915006    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6" event={"ID":"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce","Type":"ContainerStarted","Data":"0fc793365b0c48daf0bcdd5828f1cb701a918b3fcfeee294e39aa1d54fb2dddd"}
Oct 30 01:25:05.670844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:05.670789    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:05.923679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:05.923596    4576 generic.go:356] "Generic (PLEG): container finished" podID="92f04712-8b58-4e3d-9ac0-f415dbe3f7ce" containerID="3d6c465aa2bec992291e39016ff3d093ec692ecae6d2f918a12b223890c69477" exitCode=0
Oct 30 01:25:05.923679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:05.923659    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6" event={"ID":"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce","Type":"ContainerDied","Data":"3d6c465aa2bec992291e39016ff3d093ec692ecae6d2f918a12b223890c69477"}
Oct 30 01:25:06.075417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.075385    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:06.219398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.219365    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621dfe32-294e-4eb1-9afb-919e415182d3-kube-api-access-tp4hz\" (UniqueName: \"kubernetes.io/projected/621dfe32-294e-4eb1-9afb-919e415182d3-kube-api-access-tp4hz\") pod \"621dfe32-294e-4eb1-9afb-919e415182d3\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") "
Oct 30 01:25:06.219398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.219404    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-xks-ca\") pod \"621dfe32-294e-4eb1-9afb-919e415182d3\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") "
Oct 30 01:25:06.219675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.219485    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-openshift-ca\") pod \"621dfe32-294e-4eb1-9afb-919e415182d3\" (UID: \"621dfe32-294e-4eb1-9afb-919e415182d3\") "
Oct 30 01:25:06.219896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.219866    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-openshift-ca" pod "621dfe32-294e-4eb1-9afb-919e415182d3" (UID: "621dfe32-294e-4eb1-9afb-919e415182d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:06.220012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.219971    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.221749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.221724    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-xks-ca" pod "621dfe32-294e-4eb1-9afb-919e415182d3" (UID: "621dfe32-294e-4eb1-9afb-919e415182d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:06.221936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.221915    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621dfe32-294e-4eb1-9afb-919e415182d3-kube-api-access-tp4hz" pod "621dfe32-294e-4eb1-9afb-919e415182d3" (UID: "621dfe32-294e-4eb1-9afb-919e415182d3"). InnerVolumeSpecName "kube-api-access-tp4hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:06.321604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.321567    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp4hz\" (UniqueName: \"kubernetes.io/projected/621dfe32-294e-4eb1-9afb-919e415182d3-kube-api-access-tp4hz\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.321604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.321599    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621dfe32-294e-4eb1-9afb-919e415182d3-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.348068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.348032    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" containerID="cri-o://fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b" gracePeriod=45
Oct 30 01:25:06.635021 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.634991    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 01:25:06.726903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.726866    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-home\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-home\") pod \"85b941c6-c18c-46bd-be7a-58b01a29b05f\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") "
Oct 30 01:25:06.727403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.726916    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b941c6-c18c-46bd-be7a-58b01a29b05f-kube-api-access-42wzq\" (UniqueName: \"kubernetes.io/projected/85b941c6-c18c-46bd-be7a-58b01a29b05f-kube-api-access-42wzq\") pod \"85b941c6-c18c-46bd-be7a-58b01a29b05f\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") "
Oct 30 01:25:06.727403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.727035    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-tmp-dir\") pod \"85b941c6-c18c-46bd-be7a-58b01a29b05f\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") "
Oct 30 01:25:06.727403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.727090    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-dshm\") pod \"85b941c6-c18c-46bd-be7a-58b01a29b05f\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") "
Oct 30 01:25:06.727403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.727119    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-model-cache\") pod \"85b941c6-c18c-46bd-be7a-58b01a29b05f\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") "
Oct 30 01:25:06.727403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.727214    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-home" pod "85b941c6-c18c-46bd-be7a-58b01a29b05f" (UID: "85b941c6-c18c-46bd-be7a-58b01a29b05f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:06.727403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.727316    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-tmp-dir" pod "85b941c6-c18c-46bd-be7a-58b01a29b05f" (UID: "85b941c6-c18c-46bd-be7a-58b01a29b05f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:06.727403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.727361    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85b941c6-c18c-46bd-be7a-58b01a29b05f-tls-certs\" (UniqueName: \"kubernetes.io/secret/85b941c6-c18c-46bd-be7a-58b01a29b05f-tls-certs\") pod \"85b941c6-c18c-46bd-be7a-58b01a29b05f\" (UID: \"85b941c6-c18c-46bd-be7a-58b01a29b05f\") "
Oct 30 01:25:06.727845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.727490    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-model-cache" pod "85b941c6-c18c-46bd-be7a-58b01a29b05f" (UID: "85b941c6-c18c-46bd-be7a-58b01a29b05f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:06.728067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.728034    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.728067 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.728063    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.728317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.728077    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.729650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.729614    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-dshm" pod "85b941c6-c18c-46bd-be7a-58b01a29b05f" (UID: "85b941c6-c18c-46bd-be7a-58b01a29b05f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:06.729650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.729617    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b941c6-c18c-46bd-be7a-58b01a29b05f-kube-api-access-42wzq" pod "85b941c6-c18c-46bd-be7a-58b01a29b05f" (UID: "85b941c6-c18c-46bd-be7a-58b01a29b05f"). InnerVolumeSpecName "kube-api-access-42wzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:06.729917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.729896    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b941c6-c18c-46bd-be7a-58b01a29b05f-tls-certs" pod "85b941c6-c18c-46bd-be7a-58b01a29b05f" (UID: "85b941c6-c18c-46bd-be7a-58b01a29b05f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:06.830138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.830044    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b941c6-c18c-46bd-be7a-58b01a29b05f-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.830138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.830088    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42wzq\" (UniqueName: \"kubernetes.io/projected/85b941c6-c18c-46bd-be7a-58b01a29b05f-kube-api-access-42wzq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.830138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.830110    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b941c6-c18c-46bd-be7a-58b01a29b05f-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:06.931661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.931626    4576 generic.go:356] "Generic (PLEG): container finished" podID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerID="fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b" exitCode=0
Oct 30 01:25:06.931855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.931673    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"
Oct 30 01:25:06.931855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.931719    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" event={"ID":"85b941c6-c18c-46bd-be7a-58b01a29b05f","Type":"ContainerDied","Data":"fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b"}
Oct 30 01:25:06.931855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.931761    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" event={"ID":"85b941c6-c18c-46bd-be7a-58b01a29b05f","Type":"ContainerDied","Data":"d203687d1be04cd34538b4fee301008802b58d37b6336325df2fda0ecfd4d724"}
Oct 30 01:25:06.931855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.931786    4576 scope.go:122] "RemoveContainer" containerID="fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b"
Oct 30 01:25:06.933343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.933325    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"
Oct 30 01:25:06.933452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.933357    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp" event={"ID":"621dfe32-294e-4eb1-9afb-919e415182d3","Type":"ContainerDied","Data":"0c256a676cd6780cd3718c2f89caa2d10f7f779689e2aaf11ed049314e80b20b"}
Oct 30 01:25:06.933452 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.933423    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c256a676cd6780cd3718c2f89caa2d10f7f779689e2aaf11ed049314e80b20b"
Oct 30 01:25:06.943468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.943446    4576 scope.go:122] "RemoveContainer" containerID="fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b"
Oct 30 01:25:06.943823 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:06.943803    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b\": container with ID starting with fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b not found: ID does not exist" containerID="fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b"
Oct 30 01:25:06.943948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.943832    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b"} err="failed to get container status \"fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b\": rpc error: code = NotFound desc = could not find container \"fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b\": container with ID starting with fd51a99b1f6820fc6d268827e6bff462e6539a738de8b6f91446a1e92e0dc61b not found: ID does not exist"
Oct 30 01:25:06.991691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.991658    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"]
Oct 30 01:25:06.998539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:06.998506    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt"]
Oct 30 01:25:07.080414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.080352    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:07.133966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.133930    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-kube-api-access-c7qp7\" (UniqueName: \"kubernetes.io/projected/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-kube-api-access-c7qp7\") pod \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") "
Oct 30 01:25:07.134178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.133989    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-openshift-ca\") pod \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") "
Oct 30 01:25:07.134283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.134211    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-xks-ca\") pod \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\" (UID: \"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce\") "
Oct 30 01:25:07.135736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.134704    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-openshift-ca" pod "92f04712-8b58-4e3d-9ac0-f415dbe3f7ce" (UID: "92f04712-8b58-4e3d-9ac0-f415dbe3f7ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:07.138901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.138877    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-kube-api-access-c7qp7" pod "92f04712-8b58-4e3d-9ac0-f415dbe3f7ce" (UID: "92f04712-8b58-4e3d-9ac0-f415dbe3f7ce"). InnerVolumeSpecName "kube-api-access-c7qp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:07.139037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.139010    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-xks-ca" pod "92f04712-8b58-4e3d-9ac0-f415dbe3f7ce" (UID: "92f04712-8b58-4e3d-9ac0-f415dbe3f7ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:07.236346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.236267    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:07.236346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.236300    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7qp7\" (UniqueName: \"kubernetes.io/projected/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-kube-api-access-c7qp7\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:07.236346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.236318    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:07.378151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.378068    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:07.603505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.603452    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-298ddc/simulator-w3-298ddc-kserve-589cbf9754-4kbgt" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:25:07.797115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.797089    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:25:07.940046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.940005    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"]
Oct 30 01:25:07.944505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.944453    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6427d671-d974-43ef-a4d8-b255bab398e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427d671-d974-43ef-a4d8-b255bab398e1-istio-podinfo\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.944679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.944538    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-istio-token\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-istio-token\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.944679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.944576    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-data\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.944679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.944663    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-certs\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.944862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.944708    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-envoy\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.944862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.944773    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-socket\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.944978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.944810    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-kube-api-access-nzss2\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-kube-api-access-nzss2\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.945039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.944999    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6427d671-d974-43ef-a4d8-b255bab398e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427d671-d974-43ef-a4d8-b255bab398e1-istiod-ca-cert\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.945101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.945043    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-credential-socket\") pod \"6427d671-d974-43ef-a4d8-b255bab398e1\" (UID: \"6427d671-d974-43ef-a4d8-b255bab398e1\") "
Oct 30 01:25:07.945932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.945349    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-certs" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:07.945932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.945676    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:07.947038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.946346    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-data" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:07.947038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.946694    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-credential-socket" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:07.947227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.947108    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6427d671-d974-43ef-a4d8-b255bab398e1-istiod-ca-cert" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:07.947227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.947179    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-socket" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:07.948910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.948881    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-istio-token" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:07.950377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.950350    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-envoy" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:07.950377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.950359    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6427d671-d974-43ef-a4d8-b255bab398e1-istio-podinfo" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:25:07.950507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.950385    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-kube-api-access-nzss2" pod "6427d671-d974-43ef-a4d8-b255bab398e1" (UID: "6427d671-d974-43ef-a4d8-b255bab398e1"). InnerVolumeSpecName "kube-api-access-nzss2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:07.950618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.950596    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"
Oct 30 01:25:07.950745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.950594    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6" event={"ID":"92f04712-8b58-4e3d-9ac0-f415dbe3f7ce","Type":"ContainerDied","Data":"0fc793365b0c48daf0bcdd5828f1cb701a918b3fcfeee294e39aa1d54fb2dddd"}
Oct 30 01:25:07.950745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.950722    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc793365b0c48daf0bcdd5828f1cb701a918b3fcfeee294e39aa1d54fb2dddd"
Oct 30 01:25:07.952059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.952034    4576 generic.go:356] "Generic (PLEG): container finished" podID="6427d671-d974-43ef-a4d8-b255bab398e1" containerID="6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a" exitCode=0
Oct 30 01:25:07.952180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.952164    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" event={"ID":"6427d671-d974-43ef-a4d8-b255bab398e1","Type":"ContainerDied","Data":"6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a"}
Oct 30 01:25:07.952411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.952208    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj" event={"ID":"6427d671-d974-43ef-a4d8-b255bab398e1","Type":"ContainerDied","Data":"0bef313924b6ec1aeab1f35e54127baa4feb2e6634fee3732583b79a4714b497"}
Oct 30 01:25:07.952411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.952302    4576 scope.go:122] "RemoveContainer" containerID="6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a"
Oct 30 01:25:07.952535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.952418    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"
Oct 30 01:25:07.972965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.972945    4576 scope.go:122] "RemoveContainer" containerID="6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a"
Oct 30 01:25:07.973284 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:07.973252    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a\": container with ID starting with 6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a not found: ID does not exist" containerID="6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a"
Oct 30 01:25:07.973388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:07.973295    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a"} err="failed to get container status \"6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a\": rpc error: code = NotFound desc = could not find container \"6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a\": container with ID starting with 6b40cc195e9afa2290a68a34f71e965bd06b0c5984875244bddb5799be30c34a not found: ID does not exist"
Oct 30 01:25:08.044442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.044412    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"]
Oct 30 01:25:08.047122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.047094    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.047122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.047126    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.047321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.047139    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.047321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.047151    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzss2\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-kube-api-access-nzss2\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.047321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.047165    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6427d671-d974-43ef-a4d8-b255bab398e1-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.047321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.047178    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6427d671-d974-43ef-a4d8-b255bab398e1-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.047321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.047189    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6427d671-d974-43ef-a4d8-b255bab398e1-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.047321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.047202    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6427d671-d974-43ef-a4d8-b255bab398e1-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:08.050472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.050450    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-d0ae0d-openshift-default-6f7cd449cc-kglmj"]
Oct 30 01:25:08.058292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.058265    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:08.108594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.108559    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" path="/var/lib/kubelet/pods/6427d671-d974-43ef-a4d8-b255bab398e1/volumes"
Oct 30 01:25:08.109027 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.109013    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" path="/var/lib/kubelet/pods/85b941c6-c18c-46bd-be7a-58b01a29b05f/volumes"
Oct 30 01:25:08.670571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.670541    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"]
Oct 30 01:25:08.683470 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.683427    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"]
Oct 30 01:25:08.690268 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.690221    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"]
Oct 30 01:25:08.776261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.776211    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:08.992422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:08.992394    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:10.288343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:10.288299    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst" podUID="7b1bc42f-9481-44bd-91a0-53f9454889c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:25:10.288774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:10.288310    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n" podUID="cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:25:10.657357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:10.657266    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:11.341674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.341639    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"]
Oct 30 01:25:11.342136 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342123    4576 cpu_manager.go:414] "containerMap: removing container" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main"
Oct 30 01:25:11.342177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342137    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main"
Oct 30 01:25:11.342177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342147    4576 cpu_manager.go:414] "containerMap: removing container" podUID="92f04712-8b58-4e3d-9ac0-f415dbe3f7ce" containerName="revoke-keys"
Oct 30 01:25:11.342177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342152    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f04712-8b58-4e3d-9ac0-f415dbe3f7ce" containerName="revoke-keys"
Oct 30 01:25:11.342177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342177    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy"
Oct 30 01:25:11.342336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342182    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy"
Oct 30 01:25:11.342336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342189    4576 cpu_manager.go:414] "containerMap: removing container" podUID="621dfe32-294e-4eb1-9afb-919e415182d3" containerName="revoke-keys"
Oct 30 01:25:11.342336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342194    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621dfe32-294e-4eb1-9afb-919e415182d3" containerName="revoke-keys"
Oct 30 01:25:11.342336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342281    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6427d671-d974-43ef-a4d8-b255bab398e1" containerName="istio-proxy"
Oct 30 01:25:11.342336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342288    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="621dfe32-294e-4eb1-9afb-919e415182d3" containerName="revoke-keys"
Oct 30 01:25:11.342336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342298    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b941c6-c18c-46bd-be7a-58b01a29b05f" containerName="main"
Oct 30 01:25:11.342336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.342305    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f04712-8b58-4e3d-9ac0-f415dbe3f7ce" containerName="revoke-keys"
Oct 30 01:25:11.345777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.345757    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.362323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.362292    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"]
Oct 30 01:25:11.372476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.372451    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:11.372616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.372501    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:25:11.491426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.491397    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.491590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.491454    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhf5\" (UniqueName: \"kubernetes.io/projected/6e26a37c-01de-48c0-9a01-7edfc400cb32-kube-api-access-rnhf5\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.491659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.491633    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.593154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.593067    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.593381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.593156    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhf5\" (UniqueName: \"kubernetes.io/projected/6e26a37c-01de-48c0-9a01-7edfc400cb32-kube-api-access-rnhf5\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.593381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.593359    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.593885 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.593865    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.595916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.595896    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.613599 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.613576    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhf5\" (UniqueName: \"kubernetes.io/projected/6e26a37c-01de-48c0-9a01-7edfc400cb32-kube-api-access-rnhf5\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.661272 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.661226    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:11.798866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.798831    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main" containerID="cri-o://606b8401af82033f47d12f7750ab511a574cd6f830b2ff09b5044d74d7b5c702" gracePeriod=45
Oct 30 01:25:11.801933 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.801900    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main" containerID="cri-o://f6b89fc2e3be30ebdef7edded0c750269d43ab21f4a7ad4f553c1b1409af3a9d" gracePeriod=45
Oct 30 01:25:11.966088 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:11.966055    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"]
Oct 30 01:25:11.967837 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:25:11.967805    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e26a37c_01de_48c0_9a01_7edfc400cb32.slice/crio-0bd7b66b1a978a87e804bb65d4cd80c161ef0616dcd08f2b533e72df6223a837 WatchSource:0}: Error finding container 0bd7b66b1a978a87e804bb65d4cd80c161ef0616dcd08f2b533e72df6223a837: Status 404 returned error can't find the container with id 0bd7b66b1a978a87e804bb65d4cd80c161ef0616dcd08f2b533e72df6223a837
Oct 30 01:25:12.002422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.002392    4576 generic.go:356] "Generic (PLEG): container finished" podID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerID="606b8401af82033f47d12f7750ab511a574cd6f830b2ff09b5044d74d7b5c702" exitCode=0
Oct 30 01:25:12.002550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.002528    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" event={"ID":"d506eb44-fd4e-4c9a-a553-6369313404a7","Type":"ContainerDied","Data":"606b8401af82033f47d12f7750ab511a574cd6f830b2ff09b5044d74d7b5c702"}
Oct 30 01:25:12.005751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.005725    4576 generic.go:356] "Generic (PLEG): container finished" podID="c650a359-b53a-4583-ae54-c634952769a4" containerID="f6b89fc2e3be30ebdef7edded0c750269d43ab21f4a7ad4f553c1b1409af3a9d" exitCode=0
Oct 30 01:25:12.005914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.005899    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" event={"ID":"c650a359-b53a-4583-ae54-c634952769a4","Type":"ContainerDied","Data":"f6b89fc2e3be30ebdef7edded0c750269d43ab21f4a7ad4f553c1b1409af3a9d"}
Oct 30 01:25:12.007165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.007139    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr" event={"ID":"6e26a37c-01de-48c0-9a01-7edfc400cb32","Type":"ContainerStarted","Data":"0bd7b66b1a978a87e804bb65d4cd80c161ef0616dcd08f2b533e72df6223a837"}
Oct 30 01:25:12.069778 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.069669    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 01:25:12.089223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.089199    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 01:25:12.202653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.202588    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c650a359-b53a-4583-ae54-c634952769a4-kube-api-access-ldhq5\" (UniqueName: \"kubernetes.io/projected/c650a359-b53a-4583-ae54-c634952769a4-kube-api-access-ldhq5\") pod \"c650a359-b53a-4583-ae54-c634952769a4\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") "
Oct 30 01:25:12.203010 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.202989    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d506eb44-fd4e-4c9a-a553-6369313404a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/d506eb44-fd4e-4c9a-a553-6369313404a7-tls-certs\") pod \"d506eb44-fd4e-4c9a-a553-6369313404a7\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") "
Oct 30 01:25:12.203375 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203348    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-model-cache\") pod \"d506eb44-fd4e-4c9a-a553-6369313404a7\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") "
Oct 30 01:25:12.203506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203439    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c650a359-b53a-4583-ae54-c634952769a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c650a359-b53a-4583-ae54-c634952769a4-tls-certs\") pod \"c650a359-b53a-4583-ae54-c634952769a4\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") "
Oct 30 01:25:12.203506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203470    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-home\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-home\") pod \"c650a359-b53a-4583-ae54-c634952769a4\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") "
Oct 30 01:25:12.203630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203518    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-tmp-dir\") pod \"c650a359-b53a-4583-ae54-c634952769a4\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") "
Oct 30 01:25:12.203630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203553    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-dshm\") pod \"d506eb44-fd4e-4c9a-a553-6369313404a7\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") "
Oct 30 01:25:12.203630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203585    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-model-cache\") pod \"c650a359-b53a-4583-ae54-c634952769a4\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") "
Oct 30 01:25:12.203797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203629    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d506eb44-fd4e-4c9a-a553-6369313404a7-kube-api-access-vtdf6\" (UniqueName: \"kubernetes.io/projected/d506eb44-fd4e-4c9a-a553-6369313404a7-kube-api-access-vtdf6\") pod \"d506eb44-fd4e-4c9a-a553-6369313404a7\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") "
Oct 30 01:25:12.203797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203673    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-tmp-dir\") pod \"d506eb44-fd4e-4c9a-a553-6369313404a7\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") "
Oct 30 01:25:12.203797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203721    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-dshm\") pod \"c650a359-b53a-4583-ae54-c634952769a4\" (UID: \"c650a359-b53a-4583-ae54-c634952769a4\") "
Oct 30 01:25:12.203963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.203863    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-home\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-home\") pod \"d506eb44-fd4e-4c9a-a553-6369313404a7\" (UID: \"d506eb44-fd4e-4c9a-a553-6369313404a7\") "
Oct 30 01:25:12.204614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.204581    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-tmp-dir" pod "c650a359-b53a-4583-ae54-c634952769a4" (UID: "c650a359-b53a-4583-ae54-c634952769a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:12.204972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.204947    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-home" pod "c650a359-b53a-4583-ae54-c634952769a4" (UID: "c650a359-b53a-4583-ae54-c634952769a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:12.205434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.205397    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-model-cache" pod "d506eb44-fd4e-4c9a-a553-6369313404a7" (UID: "d506eb44-fd4e-4c9a-a553-6369313404a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:12.205909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.205882    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-model-cache" pod "c650a359-b53a-4583-ae54-c634952769a4" (UID: "c650a359-b53a-4583-ae54-c634952769a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:12.206090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.206066    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-tmp-dir" pod "d506eb44-fd4e-4c9a-a553-6369313404a7" (UID: "d506eb44-fd4e-4c9a-a553-6369313404a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:12.206309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.206283    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-home" pod "d506eb44-fd4e-4c9a-a553-6369313404a7" (UID: "d506eb44-fd4e-4c9a-a553-6369313404a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:12.207287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.207251    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-dshm" pod "d506eb44-fd4e-4c9a-a553-6369313404a7" (UID: "d506eb44-fd4e-4c9a-a553-6369313404a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:12.207287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.207263    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d506eb44-fd4e-4c9a-a553-6369313404a7-kube-api-access-vtdf6" pod "d506eb44-fd4e-4c9a-a553-6369313404a7" (UID: "d506eb44-fd4e-4c9a-a553-6369313404a7"). InnerVolumeSpecName "kube-api-access-vtdf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:12.207807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.207784    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.207979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.207810    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.207979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.207823    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.207979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.207837    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.207979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.207851    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtdf6\" (UniqueName: \"kubernetes.io/projected/d506eb44-fd4e-4c9a-a553-6369313404a7-kube-api-access-vtdf6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.207979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.207907    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.208317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.208011    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.208317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.208044    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d506eb44-fd4e-4c9a-a553-6369313404a7-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.208440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.208414    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d506eb44-fd4e-4c9a-a553-6369313404a7-tls-certs" pod "d506eb44-fd4e-4c9a-a553-6369313404a7" (UID: "d506eb44-fd4e-4c9a-a553-6369313404a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:12.208561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.208542    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c650a359-b53a-4583-ae54-c634952769a4-kube-api-access-ldhq5" pod "c650a359-b53a-4583-ae54-c634952769a4" (UID: "c650a359-b53a-4583-ae54-c634952769a4"). InnerVolumeSpecName "kube-api-access-ldhq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:12.209837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.209797    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-dshm" pod "c650a359-b53a-4583-ae54-c634952769a4" (UID: "c650a359-b53a-4583-ae54-c634952769a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:12.210776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.210753    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c650a359-b53a-4583-ae54-c634952769a4-tls-certs" pod "c650a359-b53a-4583-ae54-c634952769a4" (UID: "c650a359-b53a-4583-ae54-c634952769a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:12.309440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.309409    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c650a359-b53a-4583-ae54-c634952769a4-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.309440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.309436    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c650a359-b53a-4583-ae54-c634952769a4-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.309701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.309448    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhq5\" (UniqueName: \"kubernetes.io/projected/c650a359-b53a-4583-ae54-c634952769a4-kube-api-access-ldhq5\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.309701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.309458    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d506eb44-fd4e-4c9a-a553-6369313404a7-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.363665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:12.363633    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:13.016454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.016422    4576 generic.go:356] "Generic (PLEG): container finished" podID="6e26a37c-01de-48c0-9a01-7edfc400cb32" containerID="896a5d5f2aeba3541ae1e1703318ff43c68b559d6eb19fbc3ebb212045fe57c2" exitCode=0
Oct 30 01:25:13.016664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.016507    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr" event={"ID":"6e26a37c-01de-48c0-9a01-7edfc400cb32","Type":"ContainerDied","Data":"896a5d5f2aeba3541ae1e1703318ff43c68b559d6eb19fbc3ebb212045fe57c2"}
Oct 30 01:25:13.017959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.017942    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"
Oct 30 01:25:13.018110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.017985    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd" event={"ID":"d506eb44-fd4e-4c9a-a553-6369313404a7","Type":"ContainerDied","Data":"4a9394f913a8df54f46d5290454fce1696251ca66e76fba5ed322749fb5ba2b8"}
Oct 30 01:25:13.018110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.018037    4576 scope.go:122] "RemoveContainer" containerID="606b8401af82033f47d12f7750ab511a574cd6f830b2ff09b5044d74d7b5c702"
Oct 30 01:25:13.019602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.019578    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d" event={"ID":"c650a359-b53a-4583-ae54-c634952769a4","Type":"ContainerDied","Data":"31dd9b061b3b32ff2eb074e42e7d4c8689f0c8862427b31af2ae37ca33784166"}
Oct 30 01:25:13.019602 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.019597    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"
Oct 30 01:25:13.037792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.037746    4576 scope.go:122] "RemoveContainer" containerID="f6b89fc2e3be30ebdef7edded0c750269d43ab21f4a7ad4f553c1b1409af3a9d"
Oct 30 01:25:13.104348 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.104301    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"]
Oct 30 01:25:13.107146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.107124    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/simulator-w2-7d2b75-kserve-66c999c468-dsg2d"]
Oct 30 01:25:13.142759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.142674    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"]
Oct 30 01:25:13.148281 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.148245    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7d2b75/premium-w2-7d2b75-kserve-5495bc6569-hzfmd"]
Oct 30 01:25:13.841278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.838669    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:13.946276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:13.946218    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:14.112183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.112104    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c650a359-b53a-4583-ae54-c634952769a4" path="/var/lib/kubelet/pods/c650a359-b53a-4583-ae54-c634952769a4/volumes"
Oct 30 01:25:14.112787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.112765    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" path="/var/lib/kubelet/pods/d506eb44-fd4e-4c9a-a553-6369313404a7/volumes"
Oct 30 01:25:14.181730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.181702    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:14.332792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.332757    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-xks-ca\") pod \"6e26a37c-01de-48c0-9a01-7edfc400cb32\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") "
Oct 30 01:25:14.332980 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.332876    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-openshift-ca\") pod \"6e26a37c-01de-48c0-9a01-7edfc400cb32\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") "
Oct 30 01:25:14.332980 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.332911    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e26a37c-01de-48c0-9a01-7edfc400cb32-kube-api-access-rnhf5\" (UniqueName: \"kubernetes.io/projected/6e26a37c-01de-48c0-9a01-7edfc400cb32-kube-api-access-rnhf5\") pod \"6e26a37c-01de-48c0-9a01-7edfc400cb32\" (UID: \"6e26a37c-01de-48c0-9a01-7edfc400cb32\") "
Oct 30 01:25:14.333310 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.333286    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-openshift-ca" pod "6e26a37c-01de-48c0-9a01-7edfc400cb32" (UID: "6e26a37c-01de-48c0-9a01-7edfc400cb32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:14.333454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.333438    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:14.335553 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.335527    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-xks-ca" pod "6e26a37c-01de-48c0-9a01-7edfc400cb32" (UID: "6e26a37c-01de-48c0-9a01-7edfc400cb32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:14.335667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.335645    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e26a37c-01de-48c0-9a01-7edfc400cb32-kube-api-access-rnhf5" pod "6e26a37c-01de-48c0-9a01-7edfc400cb32" (UID: "6e26a37c-01de-48c0-9a01-7edfc400cb32"). InnerVolumeSpecName "kube-api-access-rnhf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:14.386415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.386333    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:14.447259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.436682    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e26a37c-01de-48c0-9a01-7edfc400cb32-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:14.447259 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.436735    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnhf5\" (UniqueName: \"kubernetes.io/projected/6e26a37c-01de-48c0-9a01-7edfc400cb32-kube-api-access-rnhf5\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:14.800768 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.800728    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:14.954166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:14.954122    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:15.037892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:15.037864    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"
Oct 30 01:25:15.038044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:15.037898    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr" event={"ID":"6e26a37c-01de-48c0-9a01-7edfc400cb32","Type":"ContainerDied","Data":"0bd7b66b1a978a87e804bb65d4cd80c161ef0616dcd08f2b533e72df6223a837"}
Oct 30 01:25:15.038044 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:15.037947    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd7b66b1a978a87e804bb65d4cd80c161ef0616dcd08f2b533e72df6223a837"
Oct 30 01:25:16.809728 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:16.809691    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:18.382785 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:18.382740    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:25:18.441848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:18.441811    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:20.288756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:20.288724    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n" podUID="cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:25:20.289148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:20.288723    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst" podUID="7b1bc42f-9481-44bd-91a0-53f9454889c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:25:20.463806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:20.463763    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:21.808279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:21.808223    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:22.103163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.103115    4576 generic.go:356] "Generic (PLEG): container finished" podID="7b1bc42f-9481-44bd-91a0-53f9454889c5" containerID="e7fdb30aaf9d394afbb58facfe2c240789f51d180f3784ad8b9389bf0f84ba18" exitCode=0
Oct 30 01:25:22.106427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.105269    4576 generic.go:356] "Generic (PLEG): container finished" podID="cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" containerID="9adb3e81f02a1516725e97bd04cea725887c5d34ea367a52ae677d136b8aa4fc" exitCode=0
Oct 30 01:25:22.108712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.108622    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst" event={"ID":"7b1bc42f-9481-44bd-91a0-53f9454889c5","Type":"ContainerDied","Data":"e7fdb30aaf9d394afbb58facfe2c240789f51d180f3784ad8b9389bf0f84ba18"}
Oct 30 01:25:22.108712 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.108671    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n" event={"ID":"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93","Type":"ContainerDied","Data":"9adb3e81f02a1516725e97bd04cea725887c5d34ea367a52ae677d136b8aa4fc"}
Oct 30 01:25:22.319173 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.319100    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 01:25:22.321522 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.321501    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 01:25:22.333464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.333437    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:22.341108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.341081    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-kube-api-access-l6qbq\" (UniqueName: \"kubernetes.io/projected/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-kube-api-access-l6qbq\") pod \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\" (UID: \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\") "
Oct 30 01:25:22.341254 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.341174    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1bc42f-9481-44bd-91a0-53f9454889c5-kube-api-access-q4rsr\" (UniqueName: \"kubernetes.io/projected/7b1bc42f-9481-44bd-91a0-53f9454889c5-kube-api-access-q4rsr\") pod \"7b1bc42f-9481-44bd-91a0-53f9454889c5\" (UID: \"7b1bc42f-9481-44bd-91a0-53f9454889c5\") "
Oct 30 01:25:22.341312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.341291    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-plugins-config-volume\") pod \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\" (UID: \"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93\") "
Oct 30 01:25:22.341365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.341331    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1bc42f-9481-44bd-91a0-53f9454889c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1bc42f-9481-44bd-91a0-53f9454889c5-plugins-config-volume\") pod \"7b1bc42f-9481-44bd-91a0-53f9454889c5\" (UID: \"7b1bc42f-9481-44bd-91a0-53f9454889c5\") "
Oct 30 01:25:22.341698 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.341674    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-plugins-config-volume" pod "cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" (UID: "cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:22.341793 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.341752    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1bc42f-9481-44bd-91a0-53f9454889c5-plugins-config-volume" pod "7b1bc42f-9481-44bd-91a0-53f9454889c5" (UID: "7b1bc42f-9481-44bd-91a0-53f9454889c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:22.341837 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.341819    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:22.343360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.343334    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-kube-api-access-l6qbq" pod "cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" (UID: "cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93"). InnerVolumeSpecName "kube-api-access-l6qbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:22.343670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.343650    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1bc42f-9481-44bd-91a0-53f9454889c5-kube-api-access-q4rsr" pod "7b1bc42f-9481-44bd-91a0-53f9454889c5" (UID: "7b1bc42f-9481-44bd-91a0-53f9454889c5"). InnerVolumeSpecName "kube-api-access-q4rsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:22.444278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.442902    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6qbq\" (UniqueName: \"kubernetes.io/projected/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93-kube-api-access-l6qbq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:22.444278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.442950    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4rsr\" (UniqueName: \"kubernetes.io/projected/7b1bc42f-9481-44bd-91a0-53f9454889c5-kube-api-access-q4rsr\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:22.444278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.442966    4576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1bc42f-9481-44bd-91a0-53f9454889c5-plugins-config-volume\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:22.475302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.475269    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:22.510926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.510897    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"]
Oct 30 01:25:22.515857 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.515833    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-298ddc-zkg4g"]
Oct 30 01:25:22.623314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.601030    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:22.964637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:22.964592    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main" containerID="cri-o://7982d29ae505e402a3f49ac1d14718366a3e46b83029ea9e3bc7948f17ba0acb" gracePeriod=45
Oct 30 01:25:23.010788 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.010751    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:23.114839 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.114810    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"
Oct 30 01:25:23.115042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.114843    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst" event={"ID":"7b1bc42f-9481-44bd-91a0-53f9454889c5","Type":"ContainerDied","Data":"15ec7de214d50c4da5de40f14a9e1948eae9fc0541d9281b4368f2b502c27340"}
Oct 30 01:25:23.115042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.114921    4576 scope.go:122] "RemoveContainer" containerID="e7fdb30aaf9d394afbb58facfe2c240789f51d180f3784ad8b9389bf0f84ba18"
Oct 30 01:25:23.116400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.116381    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"
Oct 30 01:25:23.116561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.116391    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n" event={"ID":"cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93","Type":"ContainerDied","Data":"a335ed163661ef26b32b29c5e4ce00ff32cc6bcc27e76cf850b6b6027a3d6ee4"}
Oct 30 01:25:23.125246 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.125188    4576 generic.go:356] "Generic (PLEG): container finished" podID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerID="7982d29ae505e402a3f49ac1d14718366a3e46b83029ea9e3bc7948f17ba0acb" exitCode=0
Oct 30 01:25:23.125354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.125329    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" event={"ID":"2ea6a488-e59a-4891-8c76-1e12758d30b0","Type":"ContainerDied","Data":"7982d29ae505e402a3f49ac1d14718366a3e46b83029ea9e3bc7948f17ba0acb"}
Oct 30 01:25:23.127636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.127522    4576 scope.go:122] "RemoveContainer" containerID="9adb3e81f02a1516725e97bd04cea725887c5d34ea367a52ae677d136b8aa4fc"
Oct 30 01:25:23.172662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.172632    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"]
Oct 30 01:25:23.186087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.186057    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-298ddc-ff598b48c-f9v8n"]
Oct 30 01:25:23.224985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.224953    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"]
Oct 30 01:25:23.228020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.228001    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-298ddc-76bb885cfb-74vst"]
Oct 30 01:25:23.241119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.241093    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 01:25:23.251790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.251759    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea6a488-e59a-4891-8c76-1e12758d30b0-kube-api-access-686vf\" (UniqueName: \"kubernetes.io/projected/2ea6a488-e59a-4891-8c76-1e12758d30b0-kube-api-access-686vf\") pod \"2ea6a488-e59a-4891-8c76-1e12758d30b0\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") "
Oct 30 01:25:23.251957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.251841    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-dshm\") pod \"2ea6a488-e59a-4891-8c76-1e12758d30b0\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") "
Oct 30 01:25:23.251957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.251885    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea6a488-e59a-4891-8c76-1e12758d30b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea6a488-e59a-4891-8c76-1e12758d30b0-tls-certs\") pod \"2ea6a488-e59a-4891-8c76-1e12758d30b0\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") "
Oct 30 01:25:23.252077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.251963    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-tmp-dir\") pod \"2ea6a488-e59a-4891-8c76-1e12758d30b0\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") "
Oct 30 01:25:23.252077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.252058    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-model-cache\") pod \"2ea6a488-e59a-4891-8c76-1e12758d30b0\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") "
Oct 30 01:25:23.252191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.252092    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-home\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-home\") pod \"2ea6a488-e59a-4891-8c76-1e12758d30b0\" (UID: \"2ea6a488-e59a-4891-8c76-1e12758d30b0\") "
Oct 30 01:25:23.252401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.252371    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-tmp-dir" pod "2ea6a488-e59a-4891-8c76-1e12758d30b0" (UID: "2ea6a488-e59a-4891-8c76-1e12758d30b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:23.252615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.252591    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-home" pod "2ea6a488-e59a-4891-8c76-1e12758d30b0" (UID: "2ea6a488-e59a-4891-8c76-1e12758d30b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:23.252751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.252703    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:23.252751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.252728    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:23.252751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.252602    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-model-cache" pod "2ea6a488-e59a-4891-8c76-1e12758d30b0" (UID: "2ea6a488-e59a-4891-8c76-1e12758d30b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:23.254760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.254732    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea6a488-e59a-4891-8c76-1e12758d30b0-tls-certs" pod "2ea6a488-e59a-4891-8c76-1e12758d30b0" (UID: "2ea6a488-e59a-4891-8c76-1e12758d30b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:23.254760 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.254743    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-dshm" pod "2ea6a488-e59a-4891-8c76-1e12758d30b0" (UID: "2ea6a488-e59a-4891-8c76-1e12758d30b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:23.254957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.254854    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea6a488-e59a-4891-8c76-1e12758d30b0-kube-api-access-686vf" pod "2ea6a488-e59a-4891-8c76-1e12758d30b0" (UID: "2ea6a488-e59a-4891-8c76-1e12758d30b0"). InnerVolumeSpecName "kube-api-access-686vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:23.354387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.354356    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686vf\" (UniqueName: \"kubernetes.io/projected/2ea6a488-e59a-4891-8c76-1e12758d30b0-kube-api-access-686vf\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:23.354387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.354384    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:23.354387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.354393    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea6a488-e59a-4891-8c76-1e12758d30b0-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:23.354689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.354401    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea6a488-e59a-4891-8c76-1e12758d30b0-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:23.694244 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.694182    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main" containerID="cri-o://f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a" gracePeriod=45
Oct 30 01:25:23.711410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.711378    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main" containerID="cri-o://7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315" gracePeriod=45
Oct 30 01:25:23.980592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.980559    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" containerID="cri-o://7adefbd1cbf535cb4a596ec88abf2adc5fc4fc882456facb14b3b8d8b5bf8f47" gracePeriod=45
Oct 30 01:25:23.981213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:23.981192    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 01:25:24.063148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.063109    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-model-cache\") pod \"6ad4cd2d-6793-4856-826b-bcb46811db0d\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") "
Oct 30 01:25:24.063393 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.063176    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad4cd2d-6793-4856-826b-bcb46811db0d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4cd2d-6793-4856-826b-bcb46811db0d-tls-certs\") pod \"6ad4cd2d-6793-4856-826b-bcb46811db0d\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") "
Oct 30 01:25:24.063393 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.063222    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad4cd2d-6793-4856-826b-bcb46811db0d-kube-api-access-d9gxq\" (UniqueName: \"kubernetes.io/projected/6ad4cd2d-6793-4856-826b-bcb46811db0d-kube-api-access-d9gxq\") pod \"6ad4cd2d-6793-4856-826b-bcb46811db0d\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") "
Oct 30 01:25:24.063393 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.063336    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-dshm\") pod \"6ad4cd2d-6793-4856-826b-bcb46811db0d\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") "
Oct 30 01:25:24.063598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.063436    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-home\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-home\") pod \"6ad4cd2d-6793-4856-826b-bcb46811db0d\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") "
Oct 30 01:25:24.063598 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.063482    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-tmp-dir\") pod \"6ad4cd2d-6793-4856-826b-bcb46811db0d\" (UID: \"6ad4cd2d-6793-4856-826b-bcb46811db0d\") "
Oct 30 01:25:24.064214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.064187    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-tmp-dir" pod "6ad4cd2d-6793-4856-826b-bcb46811db0d" (UID: "6ad4cd2d-6793-4856-826b-bcb46811db0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.064705 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.064412    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-model-cache" pod "6ad4cd2d-6793-4856-826b-bcb46811db0d" (UID: "6ad4cd2d-6793-4856-826b-bcb46811db0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.064959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.064935    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-home" pod "6ad4cd2d-6793-4856-826b-bcb46811db0d" (UID: "6ad4cd2d-6793-4856-826b-bcb46811db0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.067383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.067353    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-dshm" pod "6ad4cd2d-6793-4856-826b-bcb46811db0d" (UID: "6ad4cd2d-6793-4856-826b-bcb46811db0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.067511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.067395    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad4cd2d-6793-4856-826b-bcb46811db0d-tls-certs" pod "6ad4cd2d-6793-4856-826b-bcb46811db0d" (UID: "6ad4cd2d-6793-4856-826b-bcb46811db0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:24.067671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.067652    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad4cd2d-6793-4856-826b-bcb46811db0d-kube-api-access-d9gxq" pod "6ad4cd2d-6793-4856-826b-bcb46811db0d" (UID: "6ad4cd2d-6793-4856-826b-bcb46811db0d"). InnerVolumeSpecName "kube-api-access-d9gxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:24.074848 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.074814    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 01:25:24.108363 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.108330    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f43d1ba-b304-487e-a43b-ae0fdf3b0999" path="/var/lib/kubelet/pods/1f43d1ba-b304-487e-a43b-ae0fdf3b0999/volumes"
Oct 30 01:25:24.108831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.108810    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1bc42f-9481-44bd-91a0-53f9454889c5" path="/var/lib/kubelet/pods/7b1bc42f-9481-44bd-91a0-53f9454889c5/volumes"
Oct 30 01:25:24.109338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.109310    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" path="/var/lib/kubelet/pods/cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93/volumes"
Oct 30 01:25:24.147984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.147934    4576 generic.go:356] "Generic (PLEG): container finished" podID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerID="7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315" exitCode=0
Oct 30 01:25:24.148190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.148025    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" event={"ID":"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b","Type":"ContainerDied","Data":"7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315"}
Oct 30 01:25:24.148190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.148065    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx" event={"ID":"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b","Type":"ContainerDied","Data":"55f225f3a979eb4ff4dec7d10b30b5200a4498e9cbb7f933b2d0a59a0304a2e0"}
Oct 30 01:25:24.148190 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.148090    4576 scope.go:122] "RemoveContainer" containerID="7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315"
Oct 30 01:25:24.148413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.148266    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"
Oct 30 01:25:24.152302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.152270    4576 generic.go:356] "Generic (PLEG): container finished" podID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerID="f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a" exitCode=0
Oct 30 01:25:24.152425 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.152405    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" event={"ID":"6ad4cd2d-6793-4856-826b-bcb46811db0d","Type":"ContainerDied","Data":"f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a"}
Oct 30 01:25:24.152472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.152440    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb" event={"ID":"6ad4cd2d-6793-4856-826b-bcb46811db0d","Type":"ContainerDied","Data":"fd9d8277a1d286ddf2fe97d1ea4c89258e20460a9bec135e331a544eac548207"}
Oct 30 01:25:24.152524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.152511    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"
Oct 30 01:25:24.156328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.156301    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"
Oct 30 01:25:24.156695 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.156380    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4" event={"ID":"2ea6a488-e59a-4891-8c76-1e12758d30b0","Type":"ContainerDied","Data":"7aa9424af75c0d25d5d1a0e47cd5970b2480053477c7ca5705d70add8b4a833f"}
Oct 30 01:25:24.158645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.158618    4576 generic.go:356] "Generic (PLEG): container finished" podID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerID="7adefbd1cbf535cb4a596ec88abf2adc5fc4fc882456facb14b3b8d8b5bf8f47" exitCode=0
Oct 30 01:25:24.159112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.159088    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" event={"ID":"31cfa548-1c34-4604-9013-acbf1b99b3ed","Type":"ContainerDied","Data":"7adefbd1cbf535cb4a596ec88abf2adc5fc4fc882456facb14b3b8d8b5bf8f47"}
Oct 30 01:25:24.165075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.165038    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-home\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-home\") pod \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") "
Oct 30 01:25:24.165202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.165138    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-kube-api-access-smxmc\" (UniqueName: \"kubernetes.io/projected/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-kube-api-access-smxmc\") pod \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") "
Oct 30 01:25:24.165202 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.165187    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tmp-dir\") pod \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") "
Oct 30 01:25:24.165383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.165276    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-dshm\") pod \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") "
Oct 30 01:25:24.165383 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.165340    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-model-cache\") pod \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") "
Oct 30 01:25:24.165512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.165385    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tls-certs\") pod \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\" (UID: \"53955fbc-3bd0-4d7a-bd60-25f05edbcb7b\") "
Oct 30 01:25:24.165797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.165771    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-home" pod "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" (UID: "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.166357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.166331    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-model-cache" pod "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" (UID: "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.167323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.167290    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tmp-dir" pod "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" (UID: "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.168056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168026    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168052    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168063    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168075    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168086    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168185    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168204 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168201    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad4cd2d-6793-4856-826b-bcb46811db0d-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168209    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4cd2d-6793-4856-826b-bcb46811db0d-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168218    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9gxq\" (UniqueName: \"kubernetes.io/projected/6ad4cd2d-6793-4856-826b-bcb46811db0d-kube-api-access-d9gxq\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.168512 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168505    4576 scope.go:122] "RemoveContainer" containerID="7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315"
Oct 30 01:25:24.169445 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:24.168786    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315\": container with ID starting with 7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315 not found: ID does not exist" containerID="7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315"
Oct 30 01:25:24.169445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168821    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315"} err="failed to get container status \"7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315\": rpc error: code = NotFound desc = could not find container \"7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315\": container with ID starting with 7cc460a680793e872d314ba9b9071f5057791e461440e672bcbddc36e48bd315 not found: ID does not exist"
Oct 30 01:25:24.169445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.168845    4576 scope.go:122] "RemoveContainer" containerID="f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a"
Oct 30 01:25:24.169445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.169115    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": dial tcp 10.133.0.41:8000: connect: connection refused"
Oct 30 01:25:24.171806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.171771    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-kube-api-access-smxmc" pod "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" (UID: "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b"). InnerVolumeSpecName "kube-api-access-smxmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:24.172616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.172575    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-dshm" pod "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" (UID: "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.172860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.172842    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tls-certs" pod "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" (UID: "53955fbc-3bd0-4d7a-bd60-25f05edbcb7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:24.184487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.184448    4576 scope.go:122] "RemoveContainer" containerID="f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a"
Oct 30 01:25:24.184840 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:24.184817    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a\": container with ID starting with f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a not found: ID does not exist" containerID="f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a"
Oct 30 01:25:24.184908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.184853    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a"} err="failed to get container status \"f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a\": rpc error: code = NotFound desc = could not find container \"f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a\": container with ID starting with f80a08eb9e2a2371cc1d7dbe01a87f917d00773cfb4101b54f2523e57dba610a not found: ID does not exist"
Oct 30 01:25:24.184908 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.184871    4576 scope.go:122] "RemoveContainer" containerID="7982d29ae505e402a3f49ac1d14718366a3e46b83029ea9e3bc7948f17ba0acb"
Oct 30 01:25:24.205968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.205940    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"]
Oct 30 01:25:24.213380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.213321    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/distinct-w5-da76a2-kserve-6f7448ff85-zmfvb"]
Oct 30 01:25:24.252915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.252866    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"]
Oct 30 01:25:24.256832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.256804    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5a344d/distinct-2-w6-5a344d-kserve-6cfd99b645-hdnm4"]
Oct 30 01:25:24.269917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.269888    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.270062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.269921    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxmc\" (UniqueName: \"kubernetes.io/projected/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-kube-api-access-smxmc\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.270062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.269937    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.286361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.286329    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 01:25:24.370312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370277    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-home\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-home\") pod \"31cfa548-1c34-4604-9013-acbf1b99b3ed\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") "
Oct 30 01:25:24.370515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370331    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-tmp-dir\") pod \"31cfa548-1c34-4604-9013-acbf1b99b3ed\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") "
Oct 30 01:25:24.370515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370421    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31cfa548-1c34-4604-9013-acbf1b99b3ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/31cfa548-1c34-4604-9013-acbf1b99b3ed-tls-certs\") pod \"31cfa548-1c34-4604-9013-acbf1b99b3ed\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") "
Oct 30 01:25:24.370515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370508    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31cfa548-1c34-4604-9013-acbf1b99b3ed-kube-api-access-cdfc6\" (UniqueName: \"kubernetes.io/projected/31cfa548-1c34-4604-9013-acbf1b99b3ed-kube-api-access-cdfc6\") pod \"31cfa548-1c34-4604-9013-acbf1b99b3ed\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") "
Oct 30 01:25:24.370703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370579    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-dshm\") pod \"31cfa548-1c34-4604-9013-acbf1b99b3ed\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") "
Oct 30 01:25:24.370703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370582    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-home" pod "31cfa548-1c34-4604-9013-acbf1b99b3ed" (UID: "31cfa548-1c34-4604-9013-acbf1b99b3ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.370703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370615    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-model-cache\") pod \"31cfa548-1c34-4604-9013-acbf1b99b3ed\" (UID: \"31cfa548-1c34-4604-9013-acbf1b99b3ed\") "
Oct 30 01:25:24.370703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370626    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-tmp-dir" pod "31cfa548-1c34-4604-9013-acbf1b99b3ed" (UID: "31cfa548-1c34-4604-9013-acbf1b99b3ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.370983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.370954    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-model-cache" pod "31cfa548-1c34-4604-9013-acbf1b99b3ed" (UID: "31cfa548-1c34-4604-9013-acbf1b99b3ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.371127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.371074    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.371127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.371095    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.371127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.371107    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.373223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.373202    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cfa548-1c34-4604-9013-acbf1b99b3ed-tls-certs" pod "31cfa548-1c34-4604-9013-acbf1b99b3ed" (UID: "31cfa548-1c34-4604-9013-acbf1b99b3ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:24.373516 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.373494    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cfa548-1c34-4604-9013-acbf1b99b3ed-kube-api-access-cdfc6" pod "31cfa548-1c34-4604-9013-acbf1b99b3ed" (UID: "31cfa548-1c34-4604-9013-acbf1b99b3ed"). InnerVolumeSpecName "kube-api-access-cdfc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:24.374122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.374097    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-dshm" pod "31cfa548-1c34-4604-9013-acbf1b99b3ed" (UID: "31cfa548-1c34-4604-9013-acbf1b99b3ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:25:24.471916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.471848    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cfa548-1c34-4604-9013-acbf1b99b3ed-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.471916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.471876    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdfc6\" (UniqueName: \"kubernetes.io/projected/31cfa548-1c34-4604-9013-acbf1b99b3ed-kube-api-access-cdfc6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.471916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.471886    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cfa548-1c34-4604-9013-acbf1b99b3ed-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:24.476561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.476535    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:24.498492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.498462    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"]
Oct 30 01:25:24.510056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:24.510017    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/simulator-w5-da76a2-kserve-5ffd95d859-2vxpx"]
Oct 30 01:25:25.171128 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:25.171098    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"
Oct 30 01:25:25.171664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:25.171134    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n" event={"ID":"31cfa548-1c34-4604-9013-acbf1b99b3ed","Type":"ContainerDied","Data":"21d20126c0cff1231f8926584b008d0072722247827a2139e80ae71e465a165d"}
Oct 30 01:25:25.171664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:25.171183    4576 scope.go:122] "RemoveContainer" containerID="7adefbd1cbf535cb4a596ec88abf2adc5fc4fc882456facb14b3b8d8b5bf8f47"
Oct 30 01:25:25.227045 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:25.227014    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"]
Oct 30 01:25:25.240854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:25.240826    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-da76a2/premium-w5-da76a2-kserve-668bb45f74-zfp2n"]
Oct 30 01:25:25.994461 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:25.994428    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:26.116261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.115249    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" path="/var/lib/kubelet/pods/2ea6a488-e59a-4891-8c76-1e12758d30b0/volumes"
Oct 30 01:25:26.116261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.116051    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" path="/var/lib/kubelet/pods/31cfa548-1c34-4604-9013-acbf1b99b3ed/volumes"
Oct 30 01:25:26.123298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.116630    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" path="/var/lib/kubelet/pods/53955fbc-3bd0-4d7a-bd60-25f05edbcb7b/volumes"
Oct 30 01:25:26.123298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.117686    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" path="/var/lib/kubelet/pods/6ad4cd2d-6793-4856-826b-bcb46811db0d/volumes"
Oct 30 01:25:26.160277 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.159869    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:26.896756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.896723    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"]
Oct 30 01:25:26.897161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897149    4576 cpu_manager.go:414] "containerMap: removing container" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main"
Oct 30 01:25:26.897206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897162    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main"
Oct 30 01:25:26.897206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897181    4576 cpu_manager.go:414] "containerMap: removing container" podUID="7b1bc42f-9481-44bd-91a0-53f9454889c5" containerName="payload-processing"
Oct 30 01:25:26.897206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897186    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1bc42f-9481-44bd-91a0-53f9454889c5" containerName="payload-processing"
Oct 30 01:25:26.897206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897201    4576 cpu_manager.go:414] "containerMap: removing container" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main"
Oct 30 01:25:26.897206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897206    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897214    4576 cpu_manager.go:414] "containerMap: removing container" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897220    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897242    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6e26a37c-01de-48c0-9a01-7edfc400cb32" containerName="revoke-keys"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897247    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e26a37c-01de-48c0-9a01-7edfc400cb32" containerName="revoke-keys"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897256    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897260    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897267    4576 cpu_manager.go:414] "containerMap: removing container" podUID="cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" containerName="payload-pre-processing"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897276    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" containerName="payload-pre-processing"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897297    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897302    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897310    4576 cpu_manager.go:414] "containerMap: removing container" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897315    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897383    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="d506eb44-fd4e-4c9a-a553-6369313404a7" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897391    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e26a37c-01de-48c0-9a01-7edfc400cb32" containerName="revoke-keys"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897407    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad4cd2d-6793-4856-826b-bcb46811db0d" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897414    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd86ca40-96ee-41dc-ae9f-1cf43cc7bb93" containerName="payload-pre-processing"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897422    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea6a488-e59a-4891-8c76-1e12758d30b0" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897428    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="c650a359-b53a-4583-ae54-c634952769a4" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897435    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="31cfa548-1c34-4604-9013-acbf1b99b3ed" containerName="main"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897445    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1bc42f-9481-44bd-91a0-53f9454889c5" containerName="payload-processing"
Oct 30 01:25:26.897449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.897456    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="53955fbc-3bd0-4d7a-bd60-25f05edbcb7b" containerName="main"
Oct 30 01:25:26.902121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.902093    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:26.912916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.912889    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"]
Oct 30 01:25:26.917317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.917297    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:26.919151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.919122    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"]
Oct 30 01:25:26.924040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.924009    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:26.924744 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.924725    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:25:26.935356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:26.935315    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"]
Oct 30 01:25:27.003965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.003932    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8x2q\" (UniqueName: \"kubernetes.io/projected/e3831674-af14-4ae2-9721-c3055b8a9e68-kube-api-access-f8x2q\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.004165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.004048    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5w4\" (UniqueName: \"kubernetes.io/projected/02270e01-1b63-40eb-ad59-84e8efe6bec4-kube-api-access-vw5w4\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.004165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.004084    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.004165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.004146    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.004391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.004213    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.004391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.004343    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.105730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.105696    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.105923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.105769    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.105923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.105817    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x2q\" (UniqueName: \"kubernetes.io/projected/e3831674-af14-4ae2-9721-c3055b8a9e68-kube-api-access-f8x2q\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.105923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.105912    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5w4\" (UniqueName: \"kubernetes.io/projected/02270e01-1b63-40eb-ad59-84e8efe6bec4-kube-api-access-vw5w4\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.106119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.105941    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.106119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.105968    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.106670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.106646    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.106670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.106660    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.108434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.108402    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.108550 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.108402    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.127754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.127726    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x2q\" (UniqueName: \"kubernetes.io/projected/e3831674-af14-4ae2-9721-c3055b8a9e68-kube-api-access-f8x2q\") pod \"maas-api-revoke-sub-789b2252-b9mh9\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.127874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.127764    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5w4\" (UniqueName: \"kubernetes.io/projected/02270e01-1b63-40eb-ad59-84e8efe6bec4-kube-api-access-vw5w4\") pod \"maas-api-revoke-sub-bc681c33-ptkkh\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.216772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.216744    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:27.231873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.231847    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:27.297684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.297649    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:27.539414 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.537210    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"]
Oct 30 01:25:27.559838 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:25:27.559791    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02270e01_1b63_40eb_ad59_84e8efe6bec4.slice/crio-88fd676be0d6a3d35e221c8cce165604cff6ac5dde4d800e316f657411cf6380 WatchSource:0}: Error finding container 88fd676be0d6a3d35e221c8cce165604cff6ac5dde4d800e316f657411cf6380: Status 404 returned error can't find the container with id 88fd676be0d6a3d35e221c8cce165604cff6ac5dde4d800e316f657411cf6380
Oct 30 01:25:27.563565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:27.563535    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"]
Oct 30 01:25:28.051858 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.051758    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:28.281668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.280564    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9" event={"ID":"e3831674-af14-4ae2-9721-c3055b8a9e68","Type":"ContainerStarted","Data":"74b3620fea5a4cd39bb86f473c2ea883b1e2271d45f7db4400f22469d1a5b8ef"}
Oct 30 01:25:28.281668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.280632    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9" event={"ID":"e3831674-af14-4ae2-9721-c3055b8a9e68","Type":"ContainerStarted","Data":"f85346614b1b0f64a868c4923fe9b66c767ef810339d6abb0c434bd84e8aca44"}
Oct 30 01:25:28.285070 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.284366    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh" event={"ID":"02270e01-1b63-40eb-ad59-84e8efe6bec4","Type":"ContainerStarted","Data":"8198f0e88bb1df3c004140dfee8ef75c41d94f2031628210e71d93aa2d97d649"}
Oct 30 01:25:28.285070 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.284425    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh" event={"ID":"02270e01-1b63-40eb-ad59-84e8efe6bec4","Type":"ContainerStarted","Data":"88fd676be0d6a3d35e221c8cce165604cff6ac5dde4d800e316f657411cf6380"}
Oct 30 01:25:28.289455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.289255    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:28.325845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.325424    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9" podStartSLOduration=2.325398622 podStartE2EDuration="2.325398622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:28.323621223 +0000 UTC m=+2808.394945073" watchObservedRunningTime="2026-10-30 01:25:28.325398622 +0000 UTC m=+2808.396722454"
Oct 30 01:25:28.366911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.366855    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh" podStartSLOduration=2.366839374 podStartE2EDuration="2.366839374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:28.365458355 +0000 UTC m=+2808.436782204" watchObservedRunningTime="2026-10-30 01:25:28.366839374 +0000 UTC m=+2808.438163209"
Oct 30 01:25:28.787779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.787746    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"]
Oct 30 01:25:28.792019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.791994    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.810051 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.810025    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"]
Oct 30 01:25:28.833467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.833408    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz46z\" (UniqueName: \"kubernetes.io/projected/741e0095-af6d-4322-8b11-b158a24e2946-kube-api-access-vz46z\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.834153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.833564    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.834153 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.833644    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.935022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.934985    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.935203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.935075    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz46z\" (UniqueName: \"kubernetes.io/projected/741e0095-af6d-4322-8b11-b158a24e2946-kube-api-access-vz46z\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.935203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.935165    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.935680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.935660    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.937668 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.937649    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:28.955535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:28.955511    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz46z\" (UniqueName: \"kubernetes.io/projected/741e0095-af6d-4322-8b11-b158a24e2946-kube-api-access-vz46z\") pod \"maas-api-revoke-sub-66eca326-k5sds\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:29.029315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.029275    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"]
Oct 30 01:25:29.032952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.032923    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df5ef4a-sg8nk"]
Oct 30 01:25:29.114128 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.114048    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:29.191316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.188462    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:29.312722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.311664    4576 generic.go:356] "Generic (PLEG): container finished" podID="02270e01-1b63-40eb-ad59-84e8efe6bec4" containerID="8198f0e88bb1df3c004140dfee8ef75c41d94f2031628210e71d93aa2d97d649" exitCode=0
Oct 30 01:25:29.312722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.311962    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh" event={"ID":"02270e01-1b63-40eb-ad59-84e8efe6bec4","Type":"ContainerDied","Data":"8198f0e88bb1df3c004140dfee8ef75c41d94f2031628210e71d93aa2d97d649"}
Oct 30 01:25:29.316098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.316073    4576 generic.go:356] "Generic (PLEG): container finished" podID="e3831674-af14-4ae2-9721-c3055b8a9e68" containerID="74b3620fea5a4cd39bb86f473c2ea883b1e2271d45f7db4400f22469d1a5b8ef" exitCode=0
Oct 30 01:25:29.316392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.316333    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9" event={"ID":"e3831674-af14-4ae2-9721-c3055b8a9e68","Type":"ContainerDied","Data":"74b3620fea5a4cd39bb86f473c2ea883b1e2271d45f7db4400f22469d1a5b8ef"}
Oct 30 01:25:29.418850 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.418772    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"]
Oct 30 01:25:29.422432 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:25:29.422401    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741e0095_af6d_4322_8b11_b158a24e2946.slice/crio-d32e2457482cf8d26f709f057f5fcffbda1c60b8d1861552a51607eb9dbc880c WatchSource:0}: Error finding container d32e2457482cf8d26f709f057f5fcffbda1c60b8d1861552a51607eb9dbc880c: Status 404 returned error can't find the container with id d32e2457482cf8d26f709f057f5fcffbda1c60b8d1861552a51607eb9dbc880c
Oct 30 01:25:29.612181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.612149    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:29.719670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.719640    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:29.939548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:29.939514    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:30.108622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.108587    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15aecdd0-cb38-43c1-94f9-dae28674ff13" path="/var/lib/kubelet/pods/15aecdd0-cb38-43c1-94f9-dae28674ff13/volumes"
Oct 30 01:25:30.324889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.324792    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds" event={"ID":"741e0095-af6d-4322-8b11-b158a24e2946","Type":"ContainerStarted","Data":"2a4a1bb2c4424dc7ea120231400701522de7a3e81709e86569919ee2f2ad4bcf"}
Oct 30 01:25:30.324889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.324860    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds" event={"ID":"741e0095-af6d-4322-8b11-b158a24e2946","Type":"ContainerStarted","Data":"d32e2457482cf8d26f709f057f5fcffbda1c60b8d1861552a51607eb9dbc880c"}
Oct 30 01:25:30.369817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.369753    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds" podStartSLOduration=2.36973465 podStartE2EDuration="2.36973465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:30.366619015 +0000 UTC m=+2810.437942851" watchObservedRunningTime="2026-10-30 01:25:30.36973465 +0000 UTC m=+2810.441058492"
Oct 30 01:25:30.489047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.489008    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:30.519771 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.519748    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:30.568364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.568329    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02270e01-1b63-40eb-ad59-84e8efe6bec4-kube-api-access-vw5w4\" (UniqueName: \"kubernetes.io/projected/02270e01-1b63-40eb-ad59-84e8efe6bec4-kube-api-access-vw5w4\") pod \"02270e01-1b63-40eb-ad59-84e8efe6bec4\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") "
Oct 30 01:25:30.568614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.568407    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-xks-ca\") pod \"02270e01-1b63-40eb-ad59-84e8efe6bec4\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") "
Oct 30 01:25:30.568614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.568445    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3831674-af14-4ae2-9721-c3055b8a9e68-kube-api-access-f8x2q\" (UniqueName: \"kubernetes.io/projected/e3831674-af14-4ae2-9721-c3055b8a9e68-kube-api-access-f8x2q\") pod \"e3831674-af14-4ae2-9721-c3055b8a9e68\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") "
Oct 30 01:25:30.568614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.568599    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-xks-ca\") pod \"e3831674-af14-4ae2-9721-c3055b8a9e68\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") "
Oct 30 01:25:30.568862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.568713    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-openshift-ca\") pod \"02270e01-1b63-40eb-ad59-84e8efe6bec4\" (UID: \"02270e01-1b63-40eb-ad59-84e8efe6bec4\") "
Oct 30 01:25:30.568862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.568750    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-openshift-ca\") pod \"e3831674-af14-4ae2-9721-c3055b8a9e68\" (UID: \"e3831674-af14-4ae2-9721-c3055b8a9e68\") "
Oct 30 01:25:30.569346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.569324    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-openshift-ca" pod "e3831674-af14-4ae2-9721-c3055b8a9e68" (UID: "e3831674-af14-4ae2-9721-c3055b8a9e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:30.570105 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.570053    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-openshift-ca" pod "02270e01-1b63-40eb-ad59-84e8efe6bec4" (UID: "02270e01-1b63-40eb-ad59-84e8efe6bec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:30.576158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.573378    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-xks-ca" pod "02270e01-1b63-40eb-ad59-84e8efe6bec4" (UID: "02270e01-1b63-40eb-ad59-84e8efe6bec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:30.576158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.574891    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-xks-ca" pod "e3831674-af14-4ae2-9721-c3055b8a9e68" (UID: "e3831674-af14-4ae2-9721-c3055b8a9e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:30.580635 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.580604    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02270e01-1b63-40eb-ad59-84e8efe6bec4-kube-api-access-vw5w4" pod "02270e01-1b63-40eb-ad59-84e8efe6bec4" (UID: "02270e01-1b63-40eb-ad59-84e8efe6bec4"). InnerVolumeSpecName "kube-api-access-vw5w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:30.585150 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.585124    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3831674-af14-4ae2-9721-c3055b8a9e68-kube-api-access-f8x2q" pod "e3831674-af14-4ae2-9721-c3055b8a9e68" (UID: "e3831674-af14-4ae2-9721-c3055b8a9e68"). InnerVolumeSpecName "kube-api-access-f8x2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:30.669665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.669624    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:30.669665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.669653    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:30.669665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.669664    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5w4\" (UniqueName: \"kubernetes.io/projected/02270e01-1b63-40eb-ad59-84e8efe6bec4-kube-api-access-vw5w4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:30.669665 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.669673    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02270e01-1b63-40eb-ad59-84e8efe6bec4-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:30.669949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.669684    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8x2q\" (UniqueName: \"kubernetes.io/projected/e3831674-af14-4ae2-9721-c3055b8a9e68-kube-api-access-f8x2q\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:30.669949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.669693    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3831674-af14-4ae2-9721-c3055b8a9e68-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:30.865941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:30.865841    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:31.333807 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:31.333774    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9" event={"ID":"e3831674-af14-4ae2-9721-c3055b8a9e68","Type":"ContainerDied","Data":"f85346614b1b0f64a868c4923fe9b66c767ef810339d6abb0c434bd84e8aca44"}
Oct 30 01:25:31.334211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:31.333839    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f85346614b1b0f64a868c4923fe9b66c767ef810339d6abb0c434bd84e8aca44"
Oct 30 01:25:31.334211 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:31.333786    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"
Oct 30 01:25:31.336276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:31.336213    4576 generic.go:356] "Generic (PLEG): container finished" podID="741e0095-af6d-4322-8b11-b158a24e2946" containerID="2a4a1bb2c4424dc7ea120231400701522de7a3e81709e86569919ee2f2ad4bcf" exitCode=0
Oct 30 01:25:31.336403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:31.336273    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds" event={"ID":"741e0095-af6d-4322-8b11-b158a24e2946","Type":"ContainerDied","Data":"2a4a1bb2c4424dc7ea120231400701522de7a3e81709e86569919ee2f2ad4bcf"}
Oct 30 01:25:31.340459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:31.338010    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"
Oct 30 01:25:31.340459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:31.338021    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh" event={"ID":"02270e01-1b63-40eb-ad59-84e8efe6bec4","Type":"ContainerDied","Data":"88fd676be0d6a3d35e221c8cce165604cff6ac5dde4d800e316f657411cf6380"}
Oct 30 01:25:31.340459 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:31.338070    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88fd676be0d6a3d35e221c8cce165604cff6ac5dde4d800e316f657411cf6380"
Oct 30 01:25:32.165662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.165625    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"]
Oct 30 01:25:32.166158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.166144    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e3831674-af14-4ae2-9721-c3055b8a9e68" containerName="revoke-keys"
Oct 30 01:25:32.166221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.166160    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3831674-af14-4ae2-9721-c3055b8a9e68" containerName="revoke-keys"
Oct 30 01:25:32.166221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.166177    4576 cpu_manager.go:414] "containerMap: removing container" podUID="02270e01-1b63-40eb-ad59-84e8efe6bec4" containerName="revoke-keys"
Oct 30 01:25:32.166221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.166182    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02270e01-1b63-40eb-ad59-84e8efe6bec4" containerName="revoke-keys"
Oct 30 01:25:32.166347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.166266    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3831674-af14-4ae2-9721-c3055b8a9e68" containerName="revoke-keys"
Oct 30 01:25:32.166347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.166279    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="02270e01-1b63-40eb-ad59-84e8efe6bec4" containerName="revoke-keys"
Oct 30 01:25:32.169733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.169712    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.188532 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.188482    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"]
Oct 30 01:25:32.201271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.193309    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"]
Oct 30 01:25:32.201271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.193421    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.215174 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.215132    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"]
Oct 30 01:25:32.286503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.286468    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.286716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.286520    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pls\" (UniqueName: \"kubernetes.io/projected/e094331b-78e5-451d-80e4-92da6142ed29-kube-api-access-v2pls\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.286716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.286610    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.286824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.286729    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwdn8\" (UniqueName: \"kubernetes.io/projected/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-kube-api-access-hwdn8\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.286824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.286761    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.286936 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.286831    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.387892 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.387852    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.388313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.387969    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdn8\" (UniqueName: \"kubernetes.io/projected/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-kube-api-access-hwdn8\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.388313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.388007    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.388313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.388067    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.388313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.388120    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.388313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.388169    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pls\" (UniqueName: \"kubernetes.io/projected/e094331b-78e5-451d-80e4-92da6142ed29-kube-api-access-v2pls\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.389298 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.389267    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.389679 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.389597    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.391863 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.391835    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.392354 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.392334    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.410504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.410474    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pls\" (UniqueName: \"kubernetes.io/projected/e094331b-78e5-451d-80e4-92da6142ed29-kube-api-access-v2pls\") pod \"maas-api-revoke-sub-61fe78c1-g8f8p\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.413900 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.413844    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdn8\" (UniqueName: \"kubernetes.io/projected/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-kube-api-access-hwdn8\") pod \"maas-api-revoke-sub-8652274e-j6mgt\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.483547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.483517    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:32.493873 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.493832    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:32.516719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.516691    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:32.518262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.518203    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:32.595985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.595624    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741e0095-af6d-4322-8b11-b158a24e2946-kube-api-access-vz46z\" (UniqueName: \"kubernetes.io/projected/741e0095-af6d-4322-8b11-b158a24e2946-kube-api-access-vz46z\") pod \"741e0095-af6d-4322-8b11-b158a24e2946\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") "
Oct 30 01:25:32.595985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.595693    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-openshift-ca\") pod \"741e0095-af6d-4322-8b11-b158a24e2946\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") "
Oct 30 01:25:32.595985 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.595770    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-xks-ca\") pod \"741e0095-af6d-4322-8b11-b158a24e2946\" (UID: \"741e0095-af6d-4322-8b11-b158a24e2946\") "
Oct 30 01:25:32.597416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.597382    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-openshift-ca" pod "741e0095-af6d-4322-8b11-b158a24e2946" (UID: "741e0095-af6d-4322-8b11-b158a24e2946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:32.600790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.600762    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741e0095-af6d-4322-8b11-b158a24e2946-kube-api-access-vz46z" pod "741e0095-af6d-4322-8b11-b158a24e2946" (UID: "741e0095-af6d-4322-8b11-b158a24e2946"). InnerVolumeSpecName "kube-api-access-vz46z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:32.602159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.602122    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-xks-ca" pod "741e0095-af6d-4322-8b11-b158a24e2946" (UID: "741e0095-af6d-4322-8b11-b158a24e2946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:32.698221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.698186    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz46z\" (UniqueName: \"kubernetes.io/projected/741e0095-af6d-4322-8b11-b158a24e2946-kube-api-access-vz46z\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:32.698221 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.698218    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:32.698427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.698282    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e0095-af6d-4322-8b11-b158a24e2946-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:32.825489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.825444    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"]
Oct 30 01:25:32.825644 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:25:32.825598    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0e9a78_d9a4_499e_835a_4e20ed85db0c.slice/crio-050fcf9dfd78defee416f14717e54a91d6bc02b2aad5fdf61754761b49360112 WatchSource:0}: Error finding container 050fcf9dfd78defee416f14717e54a91d6bc02b2aad5fdf61754761b49360112: Status 404 returned error can't find the container with id 050fcf9dfd78defee416f14717e54a91d6bc02b2aad5fdf61754761b49360112
Oct 30 01:25:32.896467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:32.896439    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"]
Oct 30 01:25:32.897729 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:25:32.897697    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode094331b_78e5_451d_80e4_92da6142ed29.slice/crio-b70d176c7a2da05e937ee0229190a5491daeb5d0a3a4d4b5943540fa7c17f941 WatchSource:0}: Error finding container b70d176c7a2da05e937ee0229190a5491daeb5d0a3a4d4b5943540fa7c17f941: Status 404 returned error can't find the container with id b70d176c7a2da05e937ee0229190a5491daeb5d0a3a4d4b5943540fa7c17f941
Oct 30 01:25:33.358275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.358152    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p" event={"ID":"e094331b-78e5-451d-80e4-92da6142ed29","Type":"ContainerStarted","Data":"9b7138a2de42c2a3a8bb7f0d72fc31f725341377e2ef46c3e88632599982bce4"}
Oct 30 01:25:33.358275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.358210    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p" event={"ID":"e094331b-78e5-451d-80e4-92da6142ed29","Type":"ContainerStarted","Data":"b70d176c7a2da05e937ee0229190a5491daeb5d0a3a4d4b5943540fa7c17f941"}
Oct 30 01:25:33.359897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.359869    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"
Oct 30 01:25:33.359897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.359904    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds" event={"ID":"741e0095-af6d-4322-8b11-b158a24e2946","Type":"ContainerDied","Data":"d32e2457482cf8d26f709f057f5fcffbda1c60b8d1861552a51607eb9dbc880c"}
Oct 30 01:25:33.360099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.359952    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d32e2457482cf8d26f709f057f5fcffbda1c60b8d1861552a51607eb9dbc880c"
Oct 30 01:25:33.361733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.361706    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt" event={"ID":"8b0e9a78-d9a4-499e-835a-4e20ed85db0c","Type":"ContainerStarted","Data":"a41fe65947ccaafd576fcc8c1f08ff0644d2ba70d7961d4fd0c9642414fc5e65"}
Oct 30 01:25:33.361845 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.361763    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt" event={"ID":"8b0e9a78-d9a4-499e-835a-4e20ed85db0c","Type":"ContainerStarted","Data":"050fcf9dfd78defee416f14717e54a91d6bc02b2aad5fdf61754761b49360112"}
Oct 30 01:25:33.400881 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.400695    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p" podStartSLOduration=1.400675541 podStartE2EDuration="1.400675541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:33.397956164 +0000 UTC m=+2813.469279998" watchObservedRunningTime="2026-10-30 01:25:33.400675541 +0000 UTC m=+2813.471999376"
Oct 30 01:25:33.457723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.457667    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt" podStartSLOduration=1.457636868 podStartE2EDuration="1.457636868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:33.455195275 +0000 UTC m=+2813.526519134" watchObservedRunningTime="2026-10-30 01:25:33.457636868 +0000 UTC m=+2813.528960710"
Oct 30 01:25:33.831716 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.831680    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:33.975759 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:33.975717    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:34.100584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.097438    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:34.231308 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.214580    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:34.354720 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.354687    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:34.370909 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.370875    4576 generic.go:356] "Generic (PLEG): container finished" podID="e094331b-78e5-451d-80e4-92da6142ed29" containerID="9b7138a2de42c2a3a8bb7f0d72fc31f725341377e2ef46c3e88632599982bce4" exitCode=0
Oct 30 01:25:34.371081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.370959    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p" event={"ID":"e094331b-78e5-451d-80e4-92da6142ed29","Type":"ContainerDied","Data":"9b7138a2de42c2a3a8bb7f0d72fc31f725341377e2ef46c3e88632599982bce4"}
Oct 30 01:25:34.372645 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.372618    4576 generic.go:356] "Generic (PLEG): container finished" podID="8b0e9a78-d9a4-499e-835a-4e20ed85db0c" containerID="a41fe65947ccaafd576fcc8c1f08ff0644d2ba70d7961d4fd0c9642414fc5e65" exitCode=0
Oct 30 01:25:34.372765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.372698    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt" event={"ID":"8b0e9a78-d9a4-499e-835a-4e20ed85db0c","Type":"ContainerDied","Data":"a41fe65947ccaafd576fcc8c1f08ff0644d2ba70d7961d4fd0c9642414fc5e65"}
Oct 30 01:25:34.473615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.473580    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:34.516476 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.516405    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"]
Oct 30 01:25:34.519197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.519171    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fbe8-7bjvr"]
Oct 30 01:25:34.583693 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.583639    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:34.986223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:34.986188    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:35.540859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.540830    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:35.578660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.578634    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:35.631672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.631427    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e094331b-78e5-451d-80e4-92da6142ed29-kube-api-access-v2pls\" (UniqueName: \"kubernetes.io/projected/e094331b-78e5-451d-80e4-92da6142ed29-kube-api-access-v2pls\") pod \"e094331b-78e5-451d-80e4-92da6142ed29\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") "
Oct 30 01:25:35.631672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.631631    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-xks-ca\") pod \"e094331b-78e5-451d-80e4-92da6142ed29\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") "
Oct 30 01:25:35.631914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.631725    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-openshift-ca\") pod \"e094331b-78e5-451d-80e4-92da6142ed29\" (UID: \"e094331b-78e5-451d-80e4-92da6142ed29\") "
Oct 30 01:25:35.632806 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.632751    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-openshift-ca" pod "e094331b-78e5-451d-80e4-92da6142ed29" (UID: "e094331b-78e5-451d-80e4-92da6142ed29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:35.635170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.635142    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e094331b-78e5-451d-80e4-92da6142ed29-kube-api-access-v2pls" pod "e094331b-78e5-451d-80e4-92da6142ed29" (UID: "e094331b-78e5-451d-80e4-92da6142ed29"). InnerVolumeSpecName "kube-api-access-v2pls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:35.635399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.635374    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-xks-ca" pod "e094331b-78e5-451d-80e4-92da6142ed29" (UID: "e094331b-78e5-451d-80e4-92da6142ed29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:35.732650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.732617    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-openshift-ca\") pod \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") "
Oct 30 01:25:35.732865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.732669    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-kube-api-access-hwdn8\" (UniqueName: \"kubernetes.io/projected/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-kube-api-access-hwdn8\") pod \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") "
Oct 30 01:25:35.732865 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.732701    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-xks-ca\") pod \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\" (UID: \"8b0e9a78-d9a4-499e-835a-4e20ed85db0c\") "
Oct 30 01:25:35.733006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.732924    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:35.733006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.732937    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e094331b-78e5-451d-80e4-92da6142ed29-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:35.733006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.732947    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2pls\" (UniqueName: \"kubernetes.io/projected/e094331b-78e5-451d-80e4-92da6142ed29-kube-api-access-v2pls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:35.733006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.732992    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-openshift-ca" pod "8b0e9a78-d9a4-499e-835a-4e20ed85db0c" (UID: "8b0e9a78-d9a4-499e-835a-4e20ed85db0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:35.735049 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.735023    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-xks-ca" pod "8b0e9a78-d9a4-499e-835a-4e20ed85db0c" (UID: "8b0e9a78-d9a4-499e-835a-4e20ed85db0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:35.735327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.735299    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-kube-api-access-hwdn8" pod "8b0e9a78-d9a4-499e-835a-4e20ed85db0c" (UID: "8b0e9a78-d9a4-499e-835a-4e20ed85db0c"). InnerVolumeSpecName "kube-api-access-hwdn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:35.833999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.833961    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:35.833999 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.833995    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwdn8\" (UniqueName: \"kubernetes.io/projected/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-kube-api-access-hwdn8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:35.834209 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.834009    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0e9a78-d9a4-499e-835a-4e20ed85db0c-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:35.946611 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:35.946573    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:36.108286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:36.108253    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e26a37c-01de-48c0-9a01-7edfc400cb32" path="/var/lib/kubelet/pods/6e26a37c-01de-48c0-9a01-7edfc400cb32/volumes"
Oct 30 01:25:36.390735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:36.390622    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"
Oct 30 01:25:36.390735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:36.390657    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p" event={"ID":"e094331b-78e5-451d-80e4-92da6142ed29","Type":"ContainerDied","Data":"b70d176c7a2da05e937ee0229190a5491daeb5d0a3a4d4b5943540fa7c17f941"}
Oct 30 01:25:36.390735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:36.390714    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b70d176c7a2da05e937ee0229190a5491daeb5d0a3a4d4b5943540fa7c17f941"
Oct 30 01:25:36.392347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:36.392319    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"
Oct 30 01:25:36.392508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:36.392361    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt" event={"ID":"8b0e9a78-d9a4-499e-835a-4e20ed85db0c","Type":"ContainerDied","Data":"050fcf9dfd78defee416f14717e54a91d6bc02b2aad5fdf61754761b49360112"}
Oct 30 01:25:36.392508 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:36.392403    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="050fcf9dfd78defee416f14717e54a91d6bc02b2aad5fdf61754761b49360112"
Oct 30 01:25:36.474364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:36.474327    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:37.087786 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:37.087736    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:25:38.477288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:38.477220    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:39.880787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:39.880751    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:40.015401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:40.015358    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:40.035766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:40.035730    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"]
Oct 30 01:25:40.048923 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:40.048890    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73ed64e9-l7lqj"]
Oct 30 01:25:40.108538 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:40.108505    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f05c39-782f-4e33-8889-265331db52dd" path="/var/lib/kubelet/pods/f8f05c39-782f-4e33-8889-265331db52dd/volumes"
Oct 30 01:25:40.140224 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:40.140141    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:40.614392 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:40.614347    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:41.124819 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:41.124762    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:41.278175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:41.278136    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:41.400773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:41.400682    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:41.583599 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:41.582530    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:41.784920 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:41.784889    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:41.909316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:41.909280    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:42.064426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.064342    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"]
Oct 30 01:25:42.065031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065011    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e094331b-78e5-451d-80e4-92da6142ed29" containerName="revoke-keys"
Oct 30 01:25:42.065097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065034    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e094331b-78e5-451d-80e4-92da6142ed29" containerName="revoke-keys"
Oct 30 01:25:42.065097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065061    4576 cpu_manager.go:414] "containerMap: removing container" podUID="741e0095-af6d-4322-8b11-b158a24e2946" containerName="revoke-keys"
Oct 30 01:25:42.065097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065068    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741e0095-af6d-4322-8b11-b158a24e2946" containerName="revoke-keys"
Oct 30 01:25:42.065097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065088    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8b0e9a78-d9a4-499e-835a-4e20ed85db0c" containerName="revoke-keys"
Oct 30 01:25:42.065097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065095    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0e9a78-d9a4-499e-835a-4e20ed85db0c" containerName="revoke-keys"
Oct 30 01:25:42.065312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065182    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0e9a78-d9a4-499e-835a-4e20ed85db0c" containerName="revoke-keys"
Oct 30 01:25:42.065312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065196    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e094331b-78e5-451d-80e4-92da6142ed29" containerName="revoke-keys"
Oct 30 01:25:42.065312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.065204    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="741e0095-af6d-4322-8b11-b158a24e2946" containerName="revoke-keys"
Oct 30 01:25:42.068813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.068780    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.084371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.084349    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"]
Oct 30 01:25:42.096984 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.096939    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:42.097250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.096963    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:25:42.216883 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.216840    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.217410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.217055    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvm94\" (UniqueName: \"kubernetes.io/projected/bfe0b1a9-938e-4130-973e-cd2c3535f35d-kube-api-access-lvm94\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.217410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.217120    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.319065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.318966    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvm94\" (UniqueName: \"kubernetes.io/projected/bfe0b1a9-938e-4130-973e-cd2c3535f35d-kube-api-access-lvm94\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.319065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.319039    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.319336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.319094    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.319765 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.319733    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.321703 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.321680    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.338530 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.338498    4576 scope.go:122] "RemoveContainer" containerID="404ab372b0bf44f3dda2ee9bb8a0427bf70e3a4713288b140a8b67a2509698cf"
Oct 30 01:25:42.340216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.340187    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvm94\" (UniqueName: \"kubernetes.io/projected/bfe0b1a9-938e-4130-973e-cd2c3535f35d-kube-api-access-lvm94\") pod \"maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.359220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.359194    4576 scope.go:122] "RemoveContainer" containerID="0b25edceab866e882d324f0a7ca462c243a5c4f61288d4ced741e4abf6b3797e"
Oct 30 01:25:42.369194 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.369171    4576 scope.go:122] "RemoveContainer" containerID="1d46387863e575d26a2bf00d684332a349947d8f27bbc367e5b5aeed2a7de5b8"
Oct 30 01:25:42.381060 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.381033    4576 scope.go:122] "RemoveContainer" containerID="6951238c6d6181912a2ec116fc740f7463bb6a12901450c173aec2653cb5cb9d"
Oct 30 01:25:42.383247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.383202    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:42.721287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:42.721257    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"]
Oct 30 01:25:42.722409 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:25:42.722377    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe0b1a9_938e_4130_973e_cd2c3535f35d.slice/crio-210738881d8cc2de3973c12c034dfd9bfc218371fb2ef01aec1adaadb84a044a WatchSource:0}: Error finding container 210738881d8cc2de3973c12c034dfd9bfc218371fb2ef01aec1adaadb84a044a: Status 404 returned error can't find the container with id 210738881d8cc2de3973c12c034dfd9bfc218371fb2ef01aec1adaadb84a044a
Oct 30 01:25:43.463439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:43.463400    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl" event={"ID":"bfe0b1a9-938e-4130-973e-cd2c3535f35d","Type":"ContainerStarted","Data":"a1e15beb17bc2f80a1383d8578e5c0466e1666a3454b3cef1c5d5d66ab190a2e"}
Oct 30 01:25:43.463876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:43.463459    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl" event={"ID":"bfe0b1a9-938e-4130-973e-cd2c3535f35d","Type":"ContainerStarted","Data":"210738881d8cc2de3973c12c034dfd9bfc218371fb2ef01aec1adaadb84a044a"}
Oct 30 01:25:43.506545 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:43.506495    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl" podStartSLOduration=1.5064802290000001 podStartE2EDuration="1.506480229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:43.505206152 +0000 UTC m=+2823.576529987" watchObservedRunningTime="2026-10-30 01:25:43.506480229 +0000 UTC m=+2823.577804064"
Oct 30 01:25:44.069721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:44.069687    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:44.472129 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:44.472090    4576 generic.go:356] "Generic (PLEG): container finished" podID="bfe0b1a9-938e-4130-973e-cd2c3535f35d" containerID="a1e15beb17bc2f80a1383d8578e5c0466e1666a3454b3cef1c5d5d66ab190a2e" exitCode=0
Oct 30 01:25:44.472596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:44.472136    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl" event={"ID":"bfe0b1a9-938e-4130-973e-cd2c3535f35d","Type":"ContainerDied","Data":"a1e15beb17bc2f80a1383d8578e5c0466e1666a3454b3cef1c5d5d66ab190a2e"}
Oct 30 01:25:45.550134 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.550099    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:45.666758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.666649    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:45.696649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.696620    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:45.801751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.799823    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:45.879569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.879530    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-xks-ca\") pod \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") "
Oct 30 01:25:45.879782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.879653    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-openshift-ca\") pod \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") "
Oct 30 01:25:45.879782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.879714    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe0b1a9-938e-4130-973e-cd2c3535f35d-kube-api-access-lvm94\" (UniqueName: \"kubernetes.io/projected/bfe0b1a9-938e-4130-973e-cd2c3535f35d-kube-api-access-lvm94\") pod \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\" (UID: \"bfe0b1a9-938e-4130-973e-cd2c3535f35d\") "
Oct 30 01:25:45.880700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.880671    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-openshift-ca" pod "bfe0b1a9-938e-4130-973e-cd2c3535f35d" (UID: "bfe0b1a9-938e-4130-973e-cd2c3535f35d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:45.888743 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.888706    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe0b1a9-938e-4130-973e-cd2c3535f35d-kube-api-access-lvm94" pod "bfe0b1a9-938e-4130-973e-cd2c3535f35d" (UID: "bfe0b1a9-938e-4130-973e-cd2c3535f35d"). InnerVolumeSpecName "kube-api-access-lvm94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:45.889941 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.889891    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-xks-ca" pod "bfe0b1a9-938e-4130-973e-cd2c3535f35d" (UID: "bfe0b1a9-938e-4130-973e-cd2c3535f35d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:45.966399 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.966307    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:45.985297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.982203    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:45.985297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.982264    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0b1a9-938e-4130-973e-cd2c3535f35d-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:45.985297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:45.982281    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvm94\" (UniqueName: \"kubernetes.io/projected/bfe0b1a9-938e-4130-973e-cd2c3535f35d-kube-api-access-lvm94\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:46.175016 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:46.174975    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:46.290504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:46.290473    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:46.490030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:46.489991    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"
Oct 30 01:25:46.490245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:46.490003    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl" event={"ID":"bfe0b1a9-938e-4130-973e-cd2c3535f35d","Type":"ContainerDied","Data":"210738881d8cc2de3973c12c034dfd9bfc218371fb2ef01aec1adaadb84a044a"}
Oct 30 01:25:46.490245 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:46.490136    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210738881d8cc2de3973c12c034dfd9bfc218371fb2ef01aec1adaadb84a044a"
Oct 30 01:25:47.291553 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:47.291505    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:47.447438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:47.447392    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:47.575959 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:47.575694    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:47.934170 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:47.934079    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:48.057661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:48.057628    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:48.382759 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:48.382723    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:25:49.431830 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:49.431788    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:49.788264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:49.768255    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"]
Oct 30 01:25:49.788264 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:49.768506    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p" podUID="51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" containerName="maas-api" containerID="cri-o://c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc" gracePeriod=30
Oct 30 01:25:49.796472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:49.796127    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:50.052411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.052273    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:50.091460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.091432    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 01:25:50.169719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.169686    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"]
Oct 30 01:25:50.171453 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.171419    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:50.174147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.174102    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7d2b75-rnztt"]
Oct 30 01:25:50.232138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.232109    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-kube-api-access-tzphd\" (UniqueName: \"kubernetes.io/projected/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-kube-api-access-tzphd\") pod \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") "
Oct 30 01:25:50.232349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.232171    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-openshift-service-ca\") pod \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") "
Oct 30 01:25:50.232349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.232198    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-maas-api-tls\") pod \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") "
Oct 30 01:25:50.232349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.232273    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-metrics-certs\") pod \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\" (UID: \"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe\") "
Oct 30 01:25:50.232641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.232616    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-openshift-service-ca" pod "51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" (UID: "51aa3b15-f1c8-4b72-951e-c215c8c4e7fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:50.234505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.234477    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-metrics-certs" pod "51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" (UID: "51aa3b15-f1c8-4b72-951e-c215c8c4e7fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:50.234938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.234921    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-maas-api-tls" pod "51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" (UID: "51aa3b15-f1c8-4b72-951e-c215c8c4e7fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:50.235023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.234947    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-kube-api-access-tzphd" pod "51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" (UID: "51aa3b15-f1c8-4b72-951e-c215c8c4e7fe"). InnerVolumeSpecName "kube-api-access-tzphd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:50.333171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.333081    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-maas-api-tls\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:50.333171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.333111    4576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-metrics-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:50.333171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.333120    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzphd\" (UniqueName: \"kubernetes.io/projected/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-kube-api-access-tzphd\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:50.333171 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.333131    4576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe-openshift-service-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:25:50.521503 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.521468    4576 generic.go:356] "Generic (PLEG): container finished" podID="51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" containerID="c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc" exitCode=0
Oct 30 01:25:50.521993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.521519    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"
Oct 30 01:25:50.521993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.521553    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p" event={"ID":"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe","Type":"ContainerDied","Data":"c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc"}
Oct 30 01:25:50.521993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.521614    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p" event={"ID":"51aa3b15-f1c8-4b72-951e-c215c8c4e7fe","Type":"ContainerDied","Data":"3a78c9299878e1bcdb347a506f5f80728731bde35fc4b6e3af619ffba6960996"}
Oct 30 01:25:50.521993 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.521645    4576 scope.go:122] "RemoveContainer" containerID="c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc"
Oct 30 01:25:50.532437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.532415    4576 scope.go:122] "RemoveContainer" containerID="c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc"
Oct 30 01:25:50.532742 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:25:50.532723    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc\": container with ID starting with c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc not found: ID does not exist" containerID="c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc"
Oct 30 01:25:50.532794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.532753    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc"} err="failed to get container status \"c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc\": rpc error: code = NotFound desc = could not find container \"c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc\": container with ID starting with c261aafbf270978bc16bd58bfa2a2d38d5d0149fa4aaa36a5263ca7220938bcc not found: ID does not exist"
Oct 30 01:25:50.548205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.548172    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:50.577940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.577907    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"]
Oct 30 01:25:50.591466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.591395    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5a344d-b5d4b8f7f-6c44p"]
Oct 30 01:25:50.666931 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.666898    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:50.670271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.670248    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"]
Oct 30 01:25:50.672639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.672617    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5a344d-msgqp"]
Oct 30 01:25:50.787479 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:50.787437    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:51.180276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:51.180217    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:52.108733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.108699    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" path="/var/lib/kubelet/pods/51aa3b15-f1c8-4b72-951e-c215c8c4e7fe/volumes"
Oct 30 01:25:52.109276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.109122    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621dfe32-294e-4eb1-9afb-919e415182d3" path="/var/lib/kubelet/pods/621dfe32-294e-4eb1-9afb-919e415182d3/volumes"
Oct 30 01:25:52.109548 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.109531    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14266b3-4133-4451-8c50-6f86ddf04574" path="/var/lib/kubelet/pods/e14266b3-4133-4451-8c50-6f86ddf04574/volumes"
Oct 30 01:25:52.112657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.112628    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:52.614242 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.614186    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:52.781816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.781777    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:52.795304 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.795226    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"]
Oct 30 01:25:52.807106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.807076    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-da76a2-zzfm6"]
Oct 30 01:25:52.982764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:52.982720    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:53.298054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:53.297949    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:53.423291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:53.423261    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:54.108167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:54.108133    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f04712-8b58-4e3d-9ac0-f415dbe3f7ce" path="/var/lib/kubelet/pods/92f04712-8b58-4e3d-9ac0-f415dbe3f7ce/volumes"
Oct 30 01:25:54.218953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:54.218893    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:54.338667 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:54.338635    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:54.463102 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:54.463074    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:54.575225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:54.575192    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:54.903951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:54.903265    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:55.045600 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:55.045563    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:55.119329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:55.119286    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"]
Oct 30 01:25:55.135652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:55.135621    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-11e7a9-kmnxl"]
Oct 30 01:25:55.191049 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:55.191016    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:55.599861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:55.599786    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:56.107588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:56.107553    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe0b1a9-938e-4130-973e-cd2c3535f35d" path="/var/lib/kubelet/pods/bfe0b1a9-938e-4130-973e-cd2c3535f35d/volumes"
Oct 30 01:25:56.993792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:56.993753    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:58.128844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:58.128808    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:58.238626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:58.238589    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:58.430450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:58.430358    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"]
Oct 30 01:25:58.430736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:58.430683    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy" containerID="cri-o://4f00ca18e461c310b6834c19b0798b6d48b9ad7772a6a5cf0eb61e7c25fb1a62" gracePeriod=30
Oct 30 01:25:58.469181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:58.469140    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:58.589042 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:58.588999    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:58.724409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:58.724370    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:59.676390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:59.676355    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:59.791919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:59.791883    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:25:59.964813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:59.964780    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"]
Oct 30 01:25:59.965048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:25:59.965022    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy" containerID="cri-o://a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57" gracePeriod=30
Oct 30 01:26:00.013661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:00.013628    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:00.889774 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:00.889728    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:01.721416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:01.721376    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:01.832762 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:01.832727    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:02.041451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.041355    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"]
Oct 30 01:26:02.041966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.041936    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" containerID="cri-o://ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e" gracePeriod=30
Oct 30 01:26:02.077273 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.077222    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:02.440786 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.440725    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:02.563887 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.563856    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"]
Oct 30 01:26:02.564493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.564469    4576 cpu_manager.go:414] "containerMap: removing container" podUID="bfe0b1a9-938e-4130-973e-cd2c3535f35d" containerName="revoke-keys"
Oct 30 01:26:02.564493 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.564493    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe0b1a9-938e-4130-973e-cd2c3535f35d" containerName="revoke-keys"
Oct 30 01:26:02.564678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.564572    4576 cpu_manager.go:414] "containerMap: removing container" podUID="51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" containerName="maas-api"
Oct 30 01:26:02.564678 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.564582    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" containerName="maas-api"
Oct 30 01:26:02.564810 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.564793    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="51aa3b15-f1c8-4b72-951e-c215c8c4e7fe" containerName="maas-api"
Oct 30 01:26:02.564891 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.564880    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe0b1a9-938e-4130-973e-cd2c3535f35d" containerName="revoke-keys"
Oct 30 01:26:02.570048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.570019    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:02.570186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.570053    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.585146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.585016    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"]
Oct 30 01:26:02.590406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.588891    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:26:02.590640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.589375    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:26:02.590640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.589452    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:26:02.590937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.590856    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:26:02.706737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.706648    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.706737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.706694    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwm48\" (UniqueName: \"kubernetes.io/projected/36a26cd3-e83f-44b5-a0c4-469a66cf509a-kube-api-access-jwm48\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.706944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.706788    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.807701 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.807669    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.807918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.807732    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwm48\" (UniqueName: \"kubernetes.io/projected/36a26cd3-e83f-44b5-a0c4-469a66cf509a-kube-api-access-jwm48\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.807918 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.807773    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.808445 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.808416    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.810558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.810537    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.829087 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.829061    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwm48\" (UniqueName: \"kubernetes.io/projected/36a26cd3-e83f-44b5-a0c4-469a66cf509a-kube-api-access-jwm48\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:02.889381 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:02.889348    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:03.195053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.195019    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"]
Oct 30 01:26:03.196574 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:26:03.196540    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a26cd3_e83f_44b5_a0c4_469a66cf509a.slice/crio-194467ff12bfc7d51d237af158cc9ecfac925a19d653491ba56bc54648940b2e WatchSource:0}: Error finding container 194467ff12bfc7d51d237af158cc9ecfac925a19d653491ba56bc54648940b2e: Status 404 returned error can't find the container with id 194467ff12bfc7d51d237af158cc9ecfac925a19d653491ba56bc54648940b2e
Oct 30 01:26:03.619357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.619299    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg" event={"ID":"36a26cd3-e83f-44b5-a0c4-469a66cf509a","Type":"ContainerStarted","Data":"7dbaa26ed321951b38f0790b77eed1e4fab8f46467d0a1c663ef676dc47b49b9"}
Oct 30 01:26:03.619528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.619415    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg" event={"ID":"36a26cd3-e83f-44b5-a0c4-469a66cf509a","Type":"ContainerStarted","Data":"194467ff12bfc7d51d237af158cc9ecfac925a19d653491ba56bc54648940b2e"}
Oct 30 01:26:03.623799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.623760    4576 generic.go:356] "Generic (PLEG): container finished" podID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerID="4f00ca18e461c310b6834c19b0798b6d48b9ad7772a6a5cf0eb61e7c25fb1a62" exitCode=0
Oct 30 01:26:03.623966 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.623845    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" event={"ID":"9ae9e956-89af-429f-9b02-a3dcc36d9c14","Type":"ContainerDied","Data":"4f00ca18e461c310b6834c19b0798b6d48b9ad7772a6a5cf0eb61e7c25fb1a62"}
Oct 30 01:26:03.695250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.695209    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 01:26:03.742106 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.742002    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg" podStartSLOduration=1.741981848 podStartE2EDuration="1.741981848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:03.669456683 +0000 UTC m=+2843.740780519" watchObservedRunningTime="2026-10-30 01:26:03.741981848 +0000 UTC m=+2843.813305684"
Oct 30 01:26:03.820650 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820615    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-certs\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.820832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820664    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istiod-ca-cert\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.820832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820710    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-socket\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.820832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820741    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-envoy\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.820832 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820771    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-data\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.821082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820837    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-token\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.821082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820891    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-podinfo\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.821082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820939    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-kube-api-access-tjrkr\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-kube-api-access-tjrkr\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.821082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820948    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-certs" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:03.821082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.820973    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-credential-socket\") pod \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\" (UID: \"9ae9e956-89af-429f-9b02-a3dcc36d9c14\") "
Oct 30 01:26:03.821082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.821070    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istiod-ca-cert" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:03.821450 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.821114    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-socket" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:03.821506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.821443    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-credential-socket" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:03.821506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.821458    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-data" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:03.821506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.821497    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.821629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.821516    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.821629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.821530    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.823346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.823322    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-token" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:03.823531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.823499    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-kube-api-access-tjrkr" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "kube-api-access-tjrkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:03.823825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.823805    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-envoy" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:03.823890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.823874    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-podinfo" pod "9ae9e956-89af-429f-9b02-a3dcc36d9c14" (UID: "9ae9e956-89af-429f-9b02-a3dcc36d9c14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:03.922785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.922750    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.922785 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.922783    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.923007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.922794    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.923007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.922806    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae9e956-89af-429f-9b02-a3dcc36d9c14-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.923007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.922818    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrkr\" (UniqueName: \"kubernetes.io/projected/9ae9e956-89af-429f-9b02-a3dcc36d9c14-kube-api-access-tjrkr\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:03.923007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:03.922831    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae9e956-89af-429f-9b02-a3dcc36d9c14-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:04.182634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:04.182596    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:04.633860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:04.633823    4576 generic.go:356] "Generic (PLEG): container finished" podID="36a26cd3-e83f-44b5-a0c4-469a66cf509a" containerID="7dbaa26ed321951b38f0790b77eed1e4fab8f46467d0a1c663ef676dc47b49b9" exitCode=0
Oct 30 01:26:04.634388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:04.633919    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg" event={"ID":"36a26cd3-e83f-44b5-a0c4-469a66cf509a","Type":"ContainerDied","Data":"7dbaa26ed321951b38f0790b77eed1e4fab8f46467d0a1c663ef676dc47b49b9"}
Oct 30 01:26:04.635952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:04.635929    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"
Oct 30 01:26:04.635952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:04.635941    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw" event={"ID":"9ae9e956-89af-429f-9b02-a3dcc36d9c14","Type":"ContainerDied","Data":"651fcedb7dda4fae1c7b80c8fa37c9d3ea8dc6f2946b80145e459da7b09267a3"}
Oct 30 01:26:04.636177 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:04.636000    4576 scope.go:122] "RemoveContainer" containerID="4f00ca18e461c310b6834c19b0798b6d48b9ad7772a6a5cf0eb61e7c25fb1a62"
Oct 30 01:26:04.714766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:04.714727    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"]
Oct 30 01:26:04.718388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:04.718361    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5a344d-openshift-default-594fb48f99-txwqw"]
Oct 30 01:26:05.228943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.228916    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 01:26:05.338613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338525    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-envoy\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.338613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338573    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-credential-socket\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.338861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338625    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-token\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-token\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.338861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338655    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-kube-api-access-fmz2x\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-kube-api-access-fmz2x\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.338861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338730    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-certs\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.338861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338773    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-podinfo\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.338861 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338854    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-socket\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.339164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338904    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istiod-ca-cert\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.339164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.338965    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-data\") pod \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\" (UID: \"bdfe27d5-73b3-4988-91dc-87d3de6b88d5\") "
Oct 30 01:26:05.339164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.339015    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-certs" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:05.339164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.339133    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-socket" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:05.339440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.339404    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istiod-ca-cert" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:05.339440 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.339428    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-data" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:05.339714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.339446    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.339714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.339468    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.339714 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.339488    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-credential-socket" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:05.341217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.341185    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-kube-api-access-fmz2x" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "kube-api-access-fmz2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:05.341371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.341216    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-podinfo" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:05.341556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.341533    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-envoy" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:05.341556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.341539    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-token" pod "bdfe27d5-73b3-4988-91dc-87d3de6b88d5" (UID: "bdfe27d5-73b3-4988-91dc-87d3de6b88d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:05.440495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.440457    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.440495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.440489    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.440495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.440498    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmz2x\" (UniqueName: \"kubernetes.io/projected/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-kube-api-access-fmz2x\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.440757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.440509    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.440757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.440519    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.440757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.440526    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.440757 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.440534    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfe27d5-73b3-4988-91dc-87d3de6b88d5-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.644854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.644770    4576 generic.go:356] "Generic (PLEG): container finished" podID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerID="a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57" exitCode=0
Oct 30 01:26:05.644854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.644823    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"
Oct 30 01:26:05.645395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.644858    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" event={"ID":"bdfe27d5-73b3-4988-91dc-87d3de6b88d5","Type":"ContainerDied","Data":"a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57"}
Oct 30 01:26:05.645395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.644917    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q" event={"ID":"bdfe27d5-73b3-4988-91dc-87d3de6b88d5","Type":"ContainerDied","Data":"6cdaf572e08be852c5eb9e8ffd02d56562260788643a59639589018c565aa18c"}
Oct 30 01:26:05.645395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.644945    4576 scope.go:122] "RemoveContainer" containerID="a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57"
Oct 30 01:26:05.657093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.657071    4576 scope.go:122] "RemoveContainer" containerID="a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57"
Oct 30 01:26:05.657491 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:26:05.657465    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57\": container with ID starting with a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57 not found: ID does not exist" containerID="a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57"
Oct 30 01:26:05.657555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.657506    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57"} err="failed to get container status \"a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57\": rpc error: code = NotFound desc = could not find container \"a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57\": container with ID starting with a91dec50c0cdaa6a565f4d4b00cdbba94b7212f225849b5f87e9447783d37d57 not found: ID does not exist"
Oct 30 01:26:05.703568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.703541    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"]
Oct 30 01:26:05.707364 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.707300    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-da76a2-openshift-default-7f4868c97c-8wz8q"]
Oct 30 01:26:05.779028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.779002    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:05.843359 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.843318    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-xks-ca\") pod \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") "
Oct 30 01:26:05.843561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.843470    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a26cd3-e83f-44b5-a0c4-469a66cf509a-kube-api-access-jwm48\" (UniqueName: \"kubernetes.io/projected/36a26cd3-e83f-44b5-a0c4-469a66cf509a-kube-api-access-jwm48\") pod \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") "
Oct 30 01:26:05.843561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.843512    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-openshift-ca\") pod \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\" (UID: \"36a26cd3-e83f-44b5-a0c4-469a66cf509a\") "
Oct 30 01:26:05.843949 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.843922    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-openshift-ca" pod "36a26cd3-e83f-44b5-a0c4-469a66cf509a" (UID: "36a26cd3-e83f-44b5-a0c4-469a66cf509a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:05.845874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.845845    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a26cd3-e83f-44b5-a0c4-469a66cf509a-kube-api-access-jwm48" pod "36a26cd3-e83f-44b5-a0c4-469a66cf509a" (UID: "36a26cd3-e83f-44b5-a0c4-469a66cf509a"). InnerVolumeSpecName "kube-api-access-jwm48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:05.846163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.846139    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-xks-ca" pod "36a26cd3-e83f-44b5-a0c4-469a66cf509a" (UID: "36a26cd3-e83f-44b5-a0c4-469a66cf509a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:05.870970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.870942    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:05.944821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.944793    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwm48\" (UniqueName: \"kubernetes.io/projected/36a26cd3-e83f-44b5-a0c4-469a66cf509a-kube-api-access-jwm48\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.944821 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.944820    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:05.945033 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:05.944830    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a26cd3-e83f-44b5-a0c4-469a66cf509a-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:06.107563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:06.107531    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" path="/var/lib/kubelet/pods/9ae9e956-89af-429f-9b02-a3dcc36d9c14/volumes"
Oct 30 01:26:06.107981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:06.107968    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" path="/var/lib/kubelet/pods/bdfe27d5-73b3-4988-91dc-87d3de6b88d5/volumes"
Oct 30 01:26:06.654977 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:06.654947    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"
Oct 30 01:26:06.655603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:06.654984    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg" event={"ID":"36a26cd3-e83f-44b5-a0c4-469a66cf509a","Type":"ContainerDied","Data":"194467ff12bfc7d51d237af158cc9ecfac925a19d653491ba56bc54648940b2e"}
Oct 30 01:26:06.655603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:06.655040    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="194467ff12bfc7d51d237af158cc9ecfac925a19d653491ba56bc54648940b2e"
Oct 30 01:26:06.856978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:06.856943    4576 patch_prober.go:29] interesting pod/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:26:06.857213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:06.857001    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:26:07.308034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.307936    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:26:07.387458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.387428    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:07.461416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461375    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-token\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-token\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461416 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461422    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-envoy\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461451    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-credential-socket\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461531    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66bca5e0-3000-47b1-a675-e717c79a40a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bca5e0-3000-47b1-a675-e717c79a40a7-istiod-ca-cert\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461571    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-certs\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461592    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-socket\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461642    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-kube-api-access-qxcxh\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-kube-api-access-qxcxh\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461700    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-data\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461729    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-podinfo\") pod \"66bca5e0-3000-47b1-a675-e717c79a40a7\" (UID: \"66bca5e0-3000-47b1-a675-e717c79a40a7\") "
Oct 30 01:26:07.461876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461781    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-credential-socket" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:07.462112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461880    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-certs" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:07.462112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461893    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-socket" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:07.462112 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.461975    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bca5e0-3000-47b1-a675-e717c79a40a7-istiod-ca-cert" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:07.462312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.462161    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-data" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:07.462407 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.462387    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.462466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.462416    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bca5e0-3000-47b1-a675-e717c79a40a7-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.462466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.462430    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.462466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.462443    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.462595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.462480    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.463975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.463943    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-envoy" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:07.464113 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.464049    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-podinfo" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:07.464447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.464425    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-kube-api-access-qxcxh" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "kube-api-access-qxcxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:07.464447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.464436    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-token" pod "66bca5e0-3000-47b1-a675-e717c79a40a7" (UID: "66bca5e0-3000-47b1-a675-e717c79a40a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:07.563216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.563131    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.563216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.563164    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.563216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.563173    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxcxh\" (UniqueName: \"kubernetes.io/projected/66bca5e0-3000-47b1-a675-e717c79a40a7-kube-api-access-qxcxh\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.563216 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.563183    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bca5e0-3000-47b1-a675-e717c79a40a7-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:07.664840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.664806    4576 generic.go:356] "Generic (PLEG): container finished" podID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerID="ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e" exitCode=0
Oct 30 01:26:07.665350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.664867    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"
Oct 30 01:26:07.665350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.664893    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" event={"ID":"66bca5e0-3000-47b1-a675-e717c79a40a7","Type":"ContainerDied","Data":"ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e"}
Oct 30 01:26:07.665350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.664954    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt" event={"ID":"66bca5e0-3000-47b1-a675-e717c79a40a7","Type":"ContainerDied","Data":"aa761a7059d2a84e4eb3ad3ba120af7e93b681d42a9f580f349e82d6d57fa5e7"}
Oct 30 01:26:07.665350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.664987    4576 scope.go:122] "RemoveContainer" containerID="ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e"
Oct 30 01:26:07.676360 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.676319    4576 scope.go:122] "RemoveContainer" containerID="f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29"
Oct 30 01:26:07.686285 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.686265    4576 scope.go:122] "RemoveContainer" containerID="ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e"
Oct 30 01:26:07.686551 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:26:07.686532    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e\": container with ID starting with ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e not found: ID does not exist" containerID="ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e"
Oct 30 01:26:07.686629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.686561    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e"} err="failed to get container status \"ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e\": rpc error: code = NotFound desc = could not find container \"ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e\": container with ID starting with ca41acd8581a3df6c1e5335e19baca44aab011ed7023bd64c97a71787dfc9b8e not found: ID does not exist"
Oct 30 01:26:07.686629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.686579    4576 scope.go:122] "RemoveContainer" containerID="f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29"
Oct 30 01:26:07.686837 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:26:07.686819    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29\": container with ID starting with f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29 not found: ID does not exist" containerID="f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29"
Oct 30 01:26:07.686886 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.686848    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29"} err="failed to get container status \"f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29\": rpc error: code = NotFound desc = could not find container \"f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29\": container with ID starting with f46867217f1abfab7385f42feb23ab5cf6a8483d0c642825743f43a46549cc29 not found: ID does not exist"
Oct 30 01:26:07.728429 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.728394    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"]
Oct 30 01:26:07.731631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:07.731603    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-11e7a9-openshift-default-5f7f58fddf-vwpdt"]
Oct 30 01:26:08.106449 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:08.106418    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" path="/var/lib/kubelet/pods/66bca5e0-3000-47b1-a675-e717c79a40a7/volumes"
Oct 30 01:26:08.666671 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:08.666641    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:26:09.107226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:09.107190    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:10.572594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:10.572556    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:12.376150 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:12.376113    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:13.234911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:13.234867    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:13.344038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:13.344003    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:13.994163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:13.994116    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:15.995101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:15.995068    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:17.675137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:17.675105    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:18.380695 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:26:18.380651    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:26:19.174319 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:19.174283    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:20.915852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:20.915819    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:21.186193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:21.186106    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:21.295388 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:21.295356    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:21.399673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:21.399639    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:22.699158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:22.699118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:24.200162 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:24.200120    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:25.625309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:25.625256    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:26.154019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:26.153983    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:26.254040 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:26.254010    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"]
Oct 30 01:26:26.261378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:26.261347    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:26.267252 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:26.267209    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cb31e3-klwdg"]
Oct 30 01:26:26.403323 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:26.403272    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:26.783777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:26.783742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:27.996186 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:27.996151    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:28.107884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:28.107845    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a26cd3-e83f-44b5-a0c4-469a66cf509a" path="/var/lib/kubelet/pods/36a26cd3-e83f-44b5-a0c4-469a66cf509a/volumes"
Oct 30 01:26:30.092328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:30.092290    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:30.163932 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:30.163897    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:26:31.607742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:31.607713    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:32.668552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:32.668516    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:32.817521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:32.817486    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:32.944634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:32.944601    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:33.136258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.129754    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:33.305849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.305775    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"]
Oct 30 01:26:33.306025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.306001    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy" containerID="cri-o://c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b" gracePeriod=30
Oct 30 01:26:33.366108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.366078    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:33.808521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.808486    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"]
Oct 30 01:26:33.808944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.808929    4576 cpu_manager.go:414] "containerMap: removing container" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy"
Oct 30 01:26:33.808944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.808945    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy"
Oct 30 01:26:33.809041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.808975    4576 cpu_manager.go:414] "containerMap: removing container" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy"
Oct 30 01:26:33.809041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.808981    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy"
Oct 30 01:26:33.809041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.808988    4576 cpu_manager.go:414] "containerMap: removing container" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy"
Oct 30 01:26:33.809041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.808993    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy"
Oct 30 01:26:33.809041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.809008    4576 cpu_manager.go:414] "containerMap: removing container" podUID="36a26cd3-e83f-44b5-a0c4-469a66cf509a" containerName="revoke-keys"
Oct 30 01:26:33.809041 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.809013    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a26cd3-e83f-44b5-a0c4-469a66cf509a" containerName="revoke-keys"
Oct 30 01:26:33.809314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.809065    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdfe27d5-73b3-4988-91dc-87d3de6b88d5" containerName="istio-proxy"
Oct 30 01:26:33.809314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.809072    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy"
Oct 30 01:26:33.809314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.809079    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a26cd3-e83f-44b5-a0c4-469a66cf509a" containerName="revoke-keys"
Oct 30 01:26:33.809314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.809086    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae9e956-89af-429f-9b02-a3dcc36d9c14" containerName="istio-proxy"
Oct 30 01:26:33.813484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.813468    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.830120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.830091    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"]
Oct 30 01:26:33.841327 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.841298    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:26:33.841480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.841349    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:26:33.843442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.843418    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:26:33.843492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.843458    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:26:33.846371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.846340    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.846517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.846404    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcvj\" (UniqueName: \"kubernetes.io/projected/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-kube-api-access-wxcvj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.846641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.846622    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.947661 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.947631    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.947849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.947678    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.947849 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.947726    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcvj\" (UniqueName: \"kubernetes.io/projected/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-kube-api-access-wxcvj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.948288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.948268    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.950168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.950144    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:33.955185 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.955156    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:33.969093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:33.969065    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcvj\" (UniqueName: \"kubernetes.io/projected/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-kube-api-access-wxcvj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:34.126444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:34.126361    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:34.423970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:34.423941    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"]
Oct 30 01:26:34.426063 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:26:34.426035    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b4da06_5e1b_4775_bffc_88bc7a3f6e3a.slice/crio-a7d7fd3959dac31d7b1e7e7f925a00af247764ae9d58c1a805c0686dfbf1973b WatchSource:0}: Error finding container a7d7fd3959dac31d7b1e7e7f925a00af247764ae9d58c1a805c0686dfbf1973b: Status 404 returned error can't find the container with id a7d7fd3959dac31d7b1e7e7f925a00af247764ae9d58c1a805c0686dfbf1973b
Oct 30 01:26:34.859930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:34.859884    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848" event={"ID":"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a","Type":"ContainerStarted","Data":"f146b5b5f999992464c527717f1c4b88381ebd794134e25446b9c2caa4c73c67"}
Oct 30 01:26:34.860434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:34.859952    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848" event={"ID":"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a","Type":"ContainerStarted","Data":"a7d7fd3959dac31d7b1e7e7f925a00af247764ae9d58c1a805c0686dfbf1973b"}
Oct 30 01:26:34.903687 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:34.903631    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848" podStartSLOduration=1.903617348 podStartE2EDuration="1.903617348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:34.900600202 +0000 UTC m=+2874.971924038" watchObservedRunningTime="2026-10-30 01:26:34.903617348 +0000 UTC m=+2874.974941195"
Oct 30 01:26:35.666353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:35.666255    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:35.868066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:35.868030    4576 generic.go:356] "Generic (PLEG): container finished" podID="66b4da06-5e1b-4775-bffc-88bc7a3f6e3a" containerID="f146b5b5f999992464c527717f1c4b88381ebd794134e25446b9c2caa4c73c67" exitCode=0
Oct 30 01:26:35.868534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:35.868120    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848" event={"ID":"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a","Type":"ContainerDied","Data":"f146b5b5f999992464c527717f1c4b88381ebd794134e25446b9c2caa4c73c67"}
Oct 30 01:26:37.026075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.026048    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:37.079754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.079720    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-kube-api-access-wxcvj\" (UniqueName: \"kubernetes.io/projected/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-kube-api-access-wxcvj\") pod \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") "
Oct 30 01:26:37.079965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.079871    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-openshift-ca\") pod \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") "
Oct 30 01:26:37.079965 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.079902    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-xks-ca\") pod \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\" (UID: \"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a\") "
Oct 30 01:26:37.080269 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.080225    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-openshift-ca" pod "66b4da06-5e1b-4775-bffc-88bc7a3f6e3a" (UID: "66b4da06-5e1b-4775-bffc-88bc7a3f6e3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:37.082227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.082195    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-kube-api-access-wxcvj" pod "66b4da06-5e1b-4775-bffc-88bc7a3f6e3a" (UID: "66b4da06-5e1b-4775-bffc-88bc7a3f6e3a"). InnerVolumeSpecName "kube-api-access-wxcvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:37.082371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.082195    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-xks-ca" pod "66b4da06-5e1b-4775-bffc-88bc7a3f6e3a" (UID: "66b4da06-5e1b-4775-bffc-88bc7a3f6e3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:37.181286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.181187    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:37.181286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.181214    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:37.181286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.181223    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxcvj\" (UniqueName: \"kubernetes.io/projected/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a-kube-api-access-wxcvj\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:37.188756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.188721    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:37.882991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.882957    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"
Oct 30 01:26:37.882991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.882983    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848" event={"ID":"66b4da06-5e1b-4775-bffc-88bc7a3f6e3a","Type":"ContainerDied","Data":"a7d7fd3959dac31d7b1e7e7f925a00af247764ae9d58c1a805c0686dfbf1973b"}
Oct 30 01:26:37.883248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:37.883038    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7d7fd3959dac31d7b1e7e7f925a00af247764ae9d58c1a805c0686dfbf1973b"
Oct 30 01:26:38.585935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.585910    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:26:38.697585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697553    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-token\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-token\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.697787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697602    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-socket\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.697787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697638    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-kube-api-access-npjt9\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-kube-api-access-npjt9\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.697787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697730    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istiod-ca-cert\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.697787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697775    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-podinfo\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.698034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697806    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-certs\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.698034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697834    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-envoy\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.698034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697869    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-data\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.698034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697918    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-credential-socket\") pod \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\" (UID: \"9cc90602-3cba-4f39-95ff-d46de1dd32f0\") "
Oct 30 01:26:38.698034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.697975    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-socket" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:38.698352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.698210    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-certs" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:38.698352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.698225    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-data" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:38.698352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.698283    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istiod-ca-cert" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:38.698683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.698413    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.698683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.698433    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.698683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.698445    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.698683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.698454    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-credential-socket" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:38.700115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.700080    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-token" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:38.700346 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.700327    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-envoy" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:26:38.700647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.700621    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-podinfo" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:26:38.700722 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.700641    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-kube-api-access-npjt9" pod "9cc90602-3cba-4f39-95ff-d46de1dd32f0" (UID: "9cc90602-3cba-4f39-95ff-d46de1dd32f0"). InnerVolumeSpecName "kube-api-access-npjt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:38.786954 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.786923    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:38.799191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.799163    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.799191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.799186    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.799191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.799195    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.799439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.799207    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc90602-3cba-4f39-95ff-d46de1dd32f0-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.799439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.799215    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.799439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.799224    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npjt9\" (UniqueName: \"kubernetes.io/projected/9cc90602-3cba-4f39-95ff-d46de1dd32f0-kube-api-access-npjt9\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:38.892039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.891958    4576 generic.go:356] "Generic (PLEG): container finished" podID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerID="c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b" exitCode=0
Oct 30 01:26:38.892039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.892011    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"
Oct 30 01:26:38.892247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.892044    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" event={"ID":"9cc90602-3cba-4f39-95ff-d46de1dd32f0","Type":"ContainerDied","Data":"c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b"}
Oct 30 01:26:38.892247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.892106    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq" event={"ID":"9cc90602-3cba-4f39-95ff-d46de1dd32f0","Type":"ContainerDied","Data":"e63185d0afcdff8493a11fa0af27d0c2b26d3e6fd733b36fe9bf0cbdce3ac496"}
Oct 30 01:26:38.892247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.892132    4576 scope.go:122] "RemoveContainer" containerID="c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b"
Oct 30 01:26:38.903030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.903014    4576 scope.go:122] "RemoveContainer" containerID="191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a"
Oct 30 01:26:38.912504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.912487    4576 scope.go:122] "RemoveContainer" containerID="c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b"
Oct 30 01:26:38.912761 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:26:38.912745    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b\": container with ID starting with c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b not found: ID does not exist" containerID="c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b"
Oct 30 01:26:38.912842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.912767    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b"} err="failed to get container status \"c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b\": rpc error: code = NotFound desc = could not find container \"c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b\": container with ID starting with c2dd2446a5d2ce9addfedf1464ad2e1e22cead79119732759b63a0d4cc695d4b not found: ID does not exist"
Oct 30 01:26:38.912842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.912784    4576 scope.go:122] "RemoveContainer" containerID="191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a"
Oct 30 01:26:38.913032 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:26:38.913015    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a\": container with ID starting with 191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a not found: ID does not exist" containerID="191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a"
Oct 30 01:26:38.913076 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.913038    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a"} err="failed to get container status \"191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a\": rpc error: code = NotFound desc = could not find container \"191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a\": container with ID starting with 191e2f6847f429894d300d42bc61345406d00f9f9042fd0c9e6609f87bae3c4a not found: ID does not exist"
Oct 30 01:26:38.949740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.949707    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"]
Oct 30 01:26:38.951935 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:38.951915    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cb31e3-openshift-default-57bc966c5-tsdpq"]
Oct 30 01:26:40.107569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:40.107539    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" path="/var/lib/kubelet/pods/9cc90602-3cba-4f39-95ff-d46de1dd32f0/volumes"
Oct 30 01:26:40.220334 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:40.220299    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:41.804464 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:41.804433    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:42.560844 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:42.560810    4576 scope.go:122] "RemoveContainer" containerID="25584fbee8dc641dbc9d856b9afd8fbe47dafdec62a8f247975646a21b87dad2"
Oct 30 01:26:42.574699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:42.574667    4576 scope.go:122] "RemoveContainer" containerID="dda05d813602d8b8a8bd48e2893f9d5e5e7a44d80a95a81858c0568d9d938540"
Oct 30 01:26:42.584447 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:42.584421    4576 scope.go:122] "RemoveContainer" containerID="0c054868c8359f53de5aaa34f844d93f60ad65e12d38066717deed113cd8331d"
Oct 30 01:26:42.593563 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:42.593544    4576 scope.go:122] "RemoveContainer" containerID="fb48dcbd99402de8ff25a8a9c27f33030d9c3e0f033c6f603a1b9e8cf15726d6"
Oct 30 01:26:42.760852 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:42.760817    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:43.705630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:43.705596    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:44.416330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:44.416288    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:44.542812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:44.542770    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:45.222059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:45.222012    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:47.247951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:47.247910    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:47.533356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:47.533268    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:48.364573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:48.364530    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:48.382301 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:26:48.382265    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:26:48.677990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:48.677900    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:49.986180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:49.986145    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:50.538118 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:50.538078    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:50.756934 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:50.756896    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"]
Oct 30 01:26:50.770620 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:50.770592    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-ad00a0-mr848"]
Oct 30 01:26:50.771047 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:50.771020    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:50.878490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:50.878395    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:51.282438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.282405    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:51.601197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601113    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"]
Oct 30 01:26:51.601567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601552    4576 cpu_manager.go:414] "containerMap: removing container" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy"
Oct 30 01:26:51.601619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601568    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy"
Oct 30 01:26:51.601619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601592    4576 cpu_manager.go:414] "containerMap: removing container" podUID="66b4da06-5e1b-4775-bffc-88bc7a3f6e3a" containerName="revoke-keys"
Oct 30 01:26:51.601619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601597    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b4da06-5e1b-4775-bffc-88bc7a3f6e3a" containerName="revoke-keys"
Oct 30 01:26:51.601619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601611    4576 cpu_manager.go:414] "containerMap: removing container" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy"
Oct 30 01:26:51.601619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601616    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy"
Oct 30 01:26:51.601803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601673    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy"
Oct 30 01:26:51.601803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601683    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b4da06-5e1b-4775-bffc-88bc7a3f6e3a" containerName="revoke-keys"
Oct 30 01:26:51.601803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601691    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy"
Oct 30 01:26:51.601803 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.601697    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bca5e0-3000-47b1-a675-e717c79a40a7" containerName="istio-proxy"
Oct 30 01:26:51.606989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.606970    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.621629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.621601    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"]
Oct 30 01:26:51.623138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.623114    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:26:51.623299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.623198    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:26:51.657496 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.657463    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:26:51.657673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.657501    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:26:51.738480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.738442    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjl6\" (UniqueName: \"kubernetes.io/projected/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-kube-api-access-8sjl6\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.738656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.738499    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.738656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.738601    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.839441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.839408    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjl6\" (UniqueName: \"kubernetes.io/projected/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-kube-api-access-8sjl6\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.839622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.839467    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.839622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.839503    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.840483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.840455    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.841901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.841882    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.859970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.859912    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjl6\" (UniqueName: \"kubernetes.io/projected/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-kube-api-access-8sjl6\") pod \"maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:51.921056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:51.921020    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:52.107350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:52.107315    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b4da06-5e1b-4775-bffc-88bc7a3f6e3a" path="/var/lib/kubelet/pods/66b4da06-5e1b-4775-bffc-88bc7a3f6e3a/volumes"
Oct 30 01:26:52.241012 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:52.240984    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"]
Oct 30 01:26:52.242480 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:26:52.242452    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4df117b_abaf_4bbc_bf6e_3ab6137fa1ea.slice/crio-a0fefe1830342a99758f121bd56a021b88358360c907ada9fef6240d487c3bac WatchSource:0}: Error finding container a0fefe1830342a99758f121bd56a021b88358360c907ada9fef6240d487c3bac: Status 404 returned error can't find the container with id a0fefe1830342a99758f121bd56a021b88358360c907ada9fef6240d487c3bac
Oct 30 01:26:52.543928 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:52.543843    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:52.982790 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:52.982755    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2" event={"ID":"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea","Type":"ContainerStarted","Data":"38a86b1095b31a2098509db25341ad541ea551a8669e2b3fe3083fa1c160ecaf"}
Oct 30 01:26:52.982948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:52.982814    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2" event={"ID":"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea","Type":"ContainerStarted","Data":"a0fefe1830342a99758f121bd56a021b88358360c907ada9fef6240d487c3bac"}
Oct 30 01:26:53.022158 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:53.022105    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2" podStartSLOduration=2.022091759 podStartE2EDuration="2.022091759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:53.020975779 +0000 UTC m=+2893.092299616" watchObservedRunningTime="2026-10-30 01:26:53.022091759 +0000 UTC m=+2893.093415633"
Oct 30 01:26:53.990799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:53.990768    4576 generic.go:356] "Generic (PLEG): container finished" podID="f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea" containerID="38a86b1095b31a2098509db25341ad541ea551a8669e2b3fe3083fa1c160ecaf" exitCode=0
Oct 30 01:26:53.991222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:53.990863    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2" event={"ID":"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea","Type":"ContainerDied","Data":"38a86b1095b31a2098509db25341ad541ea551a8669e2b3fe3083fa1c160ecaf"}
Oct 30 01:26:54.179291 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:54.179227    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:54.246444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:54.246357    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:26:55.152491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.152459    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:55.279531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.279437    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-kube-api-access-8sjl6\" (UniqueName: \"kubernetes.io/projected/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-kube-api-access-8sjl6\") pod \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") "
Oct 30 01:26:55.279697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.279559    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-xks-ca\") pod \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") "
Oct 30 01:26:55.279697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.279664    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-openshift-ca\") pod \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\" (UID: \"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea\") "
Oct 30 01:26:55.280025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.279992    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-openshift-ca" pod "f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea" (UID: "f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:55.281855 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.281821    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-kube-api-access-8sjl6" pod "f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea" (UID: "f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea"). InnerVolumeSpecName "kube-api-access-8sjl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:55.281981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.281916    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-xks-ca" pod "f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea" (UID: "f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:55.381207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.381172    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:55.381207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.381203    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjl6\" (UniqueName: \"kubernetes.io/projected/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-kube-api-access-8sjl6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:55.381207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:55.381213    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:26:56.004397 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:56.004363    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"
Oct 30 01:26:56.004566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:56.004400    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2" event={"ID":"f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea","Type":"ContainerDied","Data":"a0fefe1830342a99758f121bd56a021b88358360c907ada9fef6240d487c3bac"}
Oct 30 01:26:56.004566 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:56.004449    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0fefe1830342a99758f121bd56a021b88358360c907ada9fef6240d487c3bac"
Oct 30 01:26:56.270043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:56.269966    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:57.279484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:57.279448    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:57.382796 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:57.382759    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:57.560621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:57.560517    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"]
Oct 30 01:26:57.560808 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:57.560779    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy" containerID="cri-o://11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b" gracePeriod=30
Oct 30 01:26:57.610737 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:57.610700    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:57.912127 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:57.912046    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:58.088674 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.088627    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:26:58.250638 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.250608    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"]
Oct 30 01:26:58.251028 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.251015    4576 cpu_manager.go:414] "containerMap: removing container" podUID="f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea" containerName="revoke-keys"
Oct 30 01:26:58.251092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.251029    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea" containerName="revoke-keys"
Oct 30 01:26:58.251092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.251052    4576 cpu_manager.go:414] "containerMap: removing container" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy"
Oct 30 01:26:58.251092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.251057    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc90602-3cba-4f39-95ff-d46de1dd32f0" containerName="istio-proxy"
Oct 30 01:26:58.251182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.251118    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea" containerName="revoke-keys"
Oct 30 01:26:58.254574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.254554    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.274466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.274439    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"]
Oct 30 01:26:58.285183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.285158    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:26:58.285565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.285185    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:26:58.285565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.285185    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:26:58.285565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.285198    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:26:58.416439 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.416407    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.416669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.416484    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6nnm\" (UniqueName: \"kubernetes.io/projected/b9bbd5b1-716f-4ae4-879d-643092107dc5-kube-api-access-r6nnm\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.416669 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.416611    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.518206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.518102    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6nnm\" (UniqueName: \"kubernetes.io/projected/b9bbd5b1-716f-4ae4-879d-643092107dc5-kube-api-access-r6nnm\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.518206 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.518172    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.518483 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.518313    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.518884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.518860    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.520791 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.520764    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.539401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.539361    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6nnm\" (UniqueName: \"kubernetes.io/projected/b9bbd5b1-716f-4ae4-879d-643092107dc5-kube-api-access-r6nnm\") pod \"maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.567690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.567657    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:26:58.867929 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:58.867904    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"]
Oct 30 01:26:58.870134 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:26:58.870095    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bbd5b1_716f_4ae4_879d_643092107dc5.slice/crio-9fc76db6a1082258f847425b3e591365e208c2705bf21945e4c14b4c2f504db1 WatchSource:0}: Error finding container 9fc76db6a1082258f847425b3e591365e208c2705bf21945e4c14b4c2f504db1: Status 404 returned error can't find the container with id 9fc76db6a1082258f847425b3e591365e208c2705bf21945e4c14b4c2f504db1
Oct 30 01:26:59.026132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:26:59.026094    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w" event={"ID":"b9bbd5b1-716f-4ae4-879d-643092107dc5","Type":"ContainerStarted","Data":"9fc76db6a1082258f847425b3e591365e208c2705bf21945e4c14b4c2f504db1"}
Oct 30 01:27:00.023983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:00.023903    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:00.034466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:00.034434    4576 generic.go:356] "Generic (PLEG): container finished" podID="b9bbd5b1-716f-4ae4-879d-643092107dc5" containerID="2c7b596b49dd202a8a316269d27d1d78408f4af074e9641813739ee481d4e488" exitCode=0
Oct 30 01:27:00.034632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:00.034522    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w" event={"ID":"b9bbd5b1-716f-4ae4-879d-643092107dc5","Type":"ContainerDied","Data":"2c7b596b49dd202a8a316269d27d1d78408f4af074e9641813739ee481d4e488"}
Oct 30 01:27:01.186723 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.186694    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:27:01.346992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.346904    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-xks-ca\") pod \"b9bbd5b1-716f-4ae4-879d-643092107dc5\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") "
Oct 30 01:27:01.346992 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.346953    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-openshift-ca\") pod \"b9bbd5b1-716f-4ae4-879d-643092107dc5\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") "
Oct 30 01:27:01.347225 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.346997    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bbd5b1-716f-4ae4-879d-643092107dc5-kube-api-access-r6nnm\" (UniqueName: \"kubernetes.io/projected/b9bbd5b1-716f-4ae4-879d-643092107dc5-kube-api-access-r6nnm\") pod \"b9bbd5b1-716f-4ae4-879d-643092107dc5\" (UID: \"b9bbd5b1-716f-4ae4-879d-643092107dc5\") "
Oct 30 01:27:01.347418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.347383    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-openshift-ca" pod "b9bbd5b1-716f-4ae4-879d-643092107dc5" (UID: "b9bbd5b1-716f-4ae4-879d-643092107dc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:01.349249 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.349211    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-xks-ca" pod "b9bbd5b1-716f-4ae4-879d-643092107dc5" (UID: "b9bbd5b1-716f-4ae4-879d-643092107dc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:01.349387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.349362    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bbd5b1-716f-4ae4-879d-643092107dc5-kube-api-access-r6nnm" pod "b9bbd5b1-716f-4ae4-879d-643092107dc5" (UID: "b9bbd5b1-716f-4ae4-879d-643092107dc5"). InnerVolumeSpecName "kube-api-access-r6nnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:01.447910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.447873    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:01.447910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.447908    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbd5b1-716f-4ae4-879d-643092107dc5-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:01.447910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.447920    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6nnm\" (UniqueName: \"kubernetes.io/projected/b9bbd5b1-716f-4ae4-879d-643092107dc5-kube-api-access-r6nnm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:01.687279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:01.687215    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:02.048213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.048126    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"
Oct 30 01:27:02.048213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.048165    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w" event={"ID":"b9bbd5b1-716f-4ae4-879d-643092107dc5","Type":"ContainerDied","Data":"9fc76db6a1082258f847425b3e591365e208c2705bf21945e4c14b4c2f504db1"}
Oct 30 01:27:02.048466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.048215    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc76db6a1082258f847425b3e591365e208c2705bf21945e4c14b4c2f504db1"
Oct 30 01:27:02.825555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.825529    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:27:02.859290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859190    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e17a171-80f1-445e-a664-b278014242c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e17a171-80f1-445e-a664-b278014242c6-istiod-ca-cert\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859259    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-data\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859297    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e17a171-80f1-445e-a664-b278014242c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e17a171-80f1-445e-a664-b278014242c6-istio-podinfo\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859337    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-certs\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859393    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-credential-socket\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859474    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-kube-api-access-2qkq8\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859506    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-envoy\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859545    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-istio-token\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-istio-token\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859572    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-socket\") pod \"6e17a171-80f1-445e-a664-b278014242c6\" (UID: \"6e17a171-80f1-445e-a664-b278014242c6\") "
Oct 30 01:27:02.859899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859653    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-data" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:02.859899 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859668    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e17a171-80f1-445e-a664-b278014242c6-istiod-ca-cert" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:02.860069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859930    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e17a171-80f1-445e-a664-b278014242c6-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:02.860069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859949    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:02.860069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859951    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-credential-socket" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:02.860069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.859963    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-socket" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:02.860338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.860308    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-certs" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:02.862009 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.861980    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-kube-api-access-2qkq8" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "kube-api-access-2qkq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:02.862009 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.861997    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e17a171-80f1-445e-a664-b278014242c6-istio-podinfo" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:27:02.862193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.862042    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-istio-token" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:02.862302 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.862214    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-envoy" pod "6e17a171-80f1-445e-a664-b278014242c6" (UID: "6e17a171-80f1-445e-a664-b278014242c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:02.961343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.961311    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e17a171-80f1-445e-a664-b278014242c6-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:02.961343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.961341    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:02.961586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.961351    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:02.961586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.961373    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkq8\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-kube-api-access-2qkq8\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:02.961586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.961383    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:02.961586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.961391    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e17a171-80f1-445e-a664-b278014242c6-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:02.961586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:02.961407    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e17a171-80f1-445e-a664-b278014242c6-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:03.042515 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.042469    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"]
Oct 30 01:27:03.057937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.057897    4576 generic.go:356] "Generic (PLEG): container finished" podID="6e17a171-80f1-445e-a664-b278014242c6" containerID="11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b" exitCode=0
Oct 30 01:27:03.058137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.058018    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"
Oct 30 01:27:03.058137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.058050    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" event={"ID":"6e17a171-80f1-445e-a664-b278014242c6","Type":"ContainerDied","Data":"11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b"}
Oct 30 01:27:03.058137 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.058132    4576 scope.go:122] "RemoveContainer" containerID="11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b"
Oct 30 01:27:03.059283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.058124    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb" event={"ID":"6e17a171-80f1-445e-a664-b278014242c6","Type":"ContainerDied","Data":"1c82d697b95eb84ee21e5f0b15b54093e1a07c444304ff22226c7b49afd31393"}
Oct 30 01:27:03.084408 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.083871    4576 scope.go:122] "RemoveContainer" containerID="11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b"
Oct 30 01:27:03.084486 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:27:03.084452    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b\": container with ID starting with 11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b not found: ID does not exist" containerID="11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b"
Oct 30 01:27:03.084526 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.084495    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b"} err="failed to get container status \"11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b\": rpc error: code = NotFound desc = could not find container \"11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b\": container with ID starting with 11fc3e4023ba7bc17aa7a1634d2869a03f08679fa1b0cf9f79714e938fd6c34b not found: ID does not exist"
Oct 30 01:27:03.130201 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.130120    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"]
Oct 30 01:27:03.132395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.132374    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-ad00a0-openshift-default-6bf9d99595-nmdjb"]
Oct 30 01:27:03.165652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.165619    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:03.276625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:03.276579    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:04.107037 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:04.107003    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e17a171-80f1-445e-a664-b278014242c6" path="/var/lib/kubelet/pods/6e17a171-80f1-445e-a664-b278014242c6/volumes"
Oct 30 01:27:04.690709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:04.690681    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:05.922581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:05.922545    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:06.069178 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:06.069115    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:06.185026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:06.184946    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:06.425260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:06.425202    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:06.545089 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:06.545041    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:06.549659 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:06.549635    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"]
Oct 30 01:27:06.560417 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:06.560395    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0519f6-8cdf2"]
Oct 30 01:27:06.662699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:06.662661    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:07.044251 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:07.044206    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:08.107520 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:08.107485    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea" path="/var/lib/kubelet/pods/f4df117b-abaf-4bbc-bf6e-3ab6137fa1ea/volumes"
Oct 30 01:27:08.288896 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:08.288863    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:08.868749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:08.868715    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:08.987055 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:08.987008    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:10.256639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:10.256606    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:11.295628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:11.295587    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:27:11.829739 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:11.829704    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:12.910710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:12.910675    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:13.028779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.028739    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:13.161825 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.161745    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"]
Oct 30 01:27:13.162148 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.162135    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy"
Oct 30 01:27:13.162279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.162148    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy"
Oct 30 01:27:13.162279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.162167    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b9bbd5b1-716f-4ae4-879d-643092107dc5" containerName="revoke-keys"
Oct 30 01:27:13.162279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.162172    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bbd5b1-716f-4ae4-879d-643092107dc5" containerName="revoke-keys"
Oct 30 01:27:13.162279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.162264    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e17a171-80f1-445e-a664-b278014242c6" containerName="istio-proxy"
Oct 30 01:27:13.162279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.162278    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bbd5b1-716f-4ae4-879d-643092107dc5" containerName="revoke-keys"
Oct 30 01:27:13.165618 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.165603    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.177614 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.177584    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"]
Oct 30 01:27:13.181309 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.181281    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.182828 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.182810    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"]
Oct 30 01:27:13.189575 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.189541    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:13.189708 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.189608    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:27:13.193835 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.193794    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:27:13.194034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.194012    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:27:13.200467 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.200442    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"]
Oct 30 01:27:13.267567 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.267537    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.267776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.267580    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.267776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.267725    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/0e9e579c-4505-4f5a-b8cd-fe1182e77177-kube-api-access-lgzl5\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.267890 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.267783    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.267929 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.267891    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfg78\" (UniqueName: \"kubernetes.io/projected/e32a2559-f34f-4062-aebf-360f1129da0a-kube-api-access-xfg78\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.267970 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.267963    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.291151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.291116    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:13.369438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.369386    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.369630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.369465    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.369630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.369529    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/0e9e579c-4505-4f5a-b8cd-fe1182e77177-kube-api-access-lgzl5\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.369630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.369580    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.369630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.369625    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfg78\" (UniqueName: \"kubernetes.io/projected/e32a2559-f34f-4062-aebf-360f1129da0a-kube-api-access-xfg78\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.369804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.369722    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.371577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.370695    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.371577 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.371413    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.374342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.374314    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.374479 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.374380    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.392975 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.392942    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/0e9e579c-4505-4f5a-b8cd-fe1182e77177-kube-api-access-lgzl5\") pod \"maas-api-revoke-sub-5cee14b6-dn4kw\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.393081 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.393013    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfg78\" (UniqueName: \"kubernetes.io/projected/e32a2559-f34f-4062-aebf-360f1129da0a-kube-api-access-xfg78\") pod \"maas-api-revoke-sub-3ceedf74-2tcpg\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.479756 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.479718    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:13.498415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.498388    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:13.527814 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.527780    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:13.801326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.801252    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"]
Oct 30 01:27:13.804486 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:27:13.804459    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9e579c_4505_4f5a_b8cd_fe1182e77177.slice/crio-c46e9edf4e757d2ba7db0cf09617a4d47fb716744af4f80eb3ce8d9bb5975f4c WatchSource:0}: Error finding container c46e9edf4e757d2ba7db0cf09617a4d47fb716744af4f80eb3ce8d9bb5975f4c: Status 404 returned error can't find the container with id c46e9edf4e757d2ba7db0cf09617a4d47fb716744af4f80eb3ce8d9bb5975f4c
Oct 30 01:27:13.831175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:13.831101    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"]
Oct 30 01:27:13.834310 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:27:13.834281    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32a2559_f34f_4062_aebf_360f1129da0a.slice/crio-d113eb13e93afde9ac23f3e69dc517c868d7b777c1ea3575c091f5f19f51b898 WatchSource:0}: Error finding container d113eb13e93afde9ac23f3e69dc517c868d7b777c1ea3575c091f5f19f51b898: Status 404 returned error can't find the container with id d113eb13e93afde9ac23f3e69dc517c868d7b777c1ea3575c091f5f19f51b898
Oct 30 01:27:14.126524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:14.126482    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw" event={"ID":"0e9e579c-4505-4f5a-b8cd-fe1182e77177","Type":"ContainerStarted","Data":"16f48f72cebded730018d40e0e9e833fee5f835c9b82aa50aaff2d6b9b3b43c9"}
Oct 30 01:27:14.127003 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:14.126546    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw" event={"ID":"0e9e579c-4505-4f5a-b8cd-fe1182e77177","Type":"ContainerStarted","Data":"c46e9edf4e757d2ba7db0cf09617a4d47fb716744af4f80eb3ce8d9bb5975f4c"}
Oct 30 01:27:14.128120 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:14.128089    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg" event={"ID":"e32a2559-f34f-4062-aebf-360f1129da0a","Type":"ContainerStarted","Data":"8397b45dd815199e25eeec2a7d1362f90393c1aed5d44a5640904c397c8c7b67"}
Oct 30 01:27:14.128222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:14.128140    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg" event={"ID":"e32a2559-f34f-4062-aebf-360f1129da0a","Type":"ContainerStarted","Data":"d113eb13e93afde9ac23f3e69dc517c868d7b777c1ea3575c091f5f19f51b898"}
Oct 30 01:27:14.168784 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:14.168730    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw" podStartSLOduration=1.168714382 podStartE2EDuration="1.168714382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:14.166313182 +0000 UTC m=+2914.237637018" watchObservedRunningTime="2026-10-30 01:27:14.168714382 +0000 UTC m=+2914.240038210"
Oct 30 01:27:14.207914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:14.207866    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg" podStartSLOduration=1.207851988 podStartE2EDuration="1.207851988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:14.205912307 +0000 UTC m=+2914.277236142" watchObservedRunningTime="2026-10-30 01:27:14.207851988 +0000 UTC m=+2914.279175823"
Oct 30 01:27:15.136163 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:15.136129    4576 generic.go:356] "Generic (PLEG): container finished" podID="e32a2559-f34f-4062-aebf-360f1129da0a" containerID="8397b45dd815199e25eeec2a7d1362f90393c1aed5d44a5640904c397c8c7b67" exitCode=0
Oct 30 01:27:15.136615 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:15.136209    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg" event={"ID":"e32a2559-f34f-4062-aebf-360f1129da0a","Type":"ContainerDied","Data":"8397b45dd815199e25eeec2a7d1362f90393c1aed5d44a5640904c397c8c7b67"}
Oct 30 01:27:15.137951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:15.137923    4576 generic.go:356] "Generic (PLEG): container finished" podID="0e9e579c-4505-4f5a-b8cd-fe1182e77177" containerID="16f48f72cebded730018d40e0e9e833fee5f835c9b82aa50aaff2d6b9b3b43c9" exitCode=0
Oct 30 01:27:15.138096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:15.137992    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw" event={"ID":"0e9e579c-4505-4f5a-b8cd-fe1182e77177","Type":"ContainerDied","Data":"16f48f72cebded730018d40e0e9e833fee5f835c9b82aa50aaff2d6b9b3b43c9"}
Oct 30 01:27:15.346147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:15.346101    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:16.287026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.287000    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:16.318349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.318326    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:16.400921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.400843    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-xks-ca\") pod \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") "
Oct 30 01:27:16.400921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.400878    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-xks-ca\") pod \"e32a2559-f34f-4062-aebf-360f1129da0a\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") "
Oct 30 01:27:16.401131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.400930    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-openshift-ca\") pod \"e32a2559-f34f-4062-aebf-360f1129da0a\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") "
Oct 30 01:27:16.401131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.401010    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32a2559-f34f-4062-aebf-360f1129da0a-kube-api-access-xfg78\" (UniqueName: \"kubernetes.io/projected/e32a2559-f34f-4062-aebf-360f1129da0a-kube-api-access-xfg78\") pod \"e32a2559-f34f-4062-aebf-360f1129da0a\" (UID: \"e32a2559-f34f-4062-aebf-360f1129da0a\") "
Oct 30 01:27:16.401131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.401055    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9e579c-4505-4f5a-b8cd-fe1182e77177-kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/0e9e579c-4505-4f5a-b8cd-fe1182e77177-kube-api-access-lgzl5\") pod \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") "
Oct 30 01:27:16.401131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.401095    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-openshift-ca\") pod \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\" (UID: \"0e9e579c-4505-4f5a-b8cd-fe1182e77177\") "
Oct 30 01:27:16.401495 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.401463    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-openshift-ca" pod "e32a2559-f34f-4062-aebf-360f1129da0a" (UID: "e32a2559-f34f-4062-aebf-360f1129da0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:16.401672 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.401589    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:16.401940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.401912    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-openshift-ca" pod "0e9e579c-4505-4f5a-b8cd-fe1182e77177" (UID: "0e9e579c-4505-4f5a-b8cd-fe1182e77177"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:16.403432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.403404    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-xks-ca" pod "e32a2559-f34f-4062-aebf-360f1129da0a" (UID: "e32a2559-f34f-4062-aebf-360f1129da0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:16.403547 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.403425    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-xks-ca" pod "0e9e579c-4505-4f5a-b8cd-fe1182e77177" (UID: "0e9e579c-4505-4f5a-b8cd-fe1182e77177"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:16.403841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.403821    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32a2559-f34f-4062-aebf-360f1129da0a-kube-api-access-xfg78" pod "e32a2559-f34f-4062-aebf-360f1129da0a" (UID: "e32a2559-f34f-4062-aebf-360f1129da0a"). InnerVolumeSpecName "kube-api-access-xfg78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:16.403911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.403835    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9e579c-4505-4f5a-b8cd-fe1182e77177-kube-api-access-lgzl5" pod "0e9e579c-4505-4f5a-b8cd-fe1182e77177" (UID: "0e9e579c-4505-4f5a-b8cd-fe1182e77177"). InnerVolumeSpecName "kube-api-access-lgzl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:16.502836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.502802    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfg78\" (UniqueName: \"kubernetes.io/projected/e32a2559-f34f-4062-aebf-360f1129da0a-kube-api-access-xfg78\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:16.502836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.502830    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgzl5\" (UniqueName: \"kubernetes.io/projected/0e9e579c-4505-4f5a-b8cd-fe1182e77177-kube-api-access-lgzl5\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:16.502836 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.502840    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:16.503091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.502849    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9e579c-4505-4f5a-b8cd-fe1182e77177-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:16.503091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.502857    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32a2559-f34f-4062-aebf-360f1129da0a-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:16.857727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:16.857692    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:17.159286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:17.159149    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw" event={"ID":"0e9e579c-4505-4f5a-b8cd-fe1182e77177","Type":"ContainerDied","Data":"c46e9edf4e757d2ba7db0cf09617a4d47fb716744af4f80eb3ce8d9bb5975f4c"}
Oct 30 01:27:17.159286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:17.159214    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46e9edf4e757d2ba7db0cf09617a4d47fb716744af4f80eb3ce8d9bb5975f4c"
Oct 30 01:27:17.159286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:17.159219    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"
Oct 30 01:27:17.160902 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:17.160866    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"
Oct 30 01:27:17.161070 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:17.160864    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg" event={"ID":"e32a2559-f34f-4062-aebf-360f1129da0a","Type":"ContainerDied","Data":"d113eb13e93afde9ac23f3e69dc517c868d7b777c1ea3575c091f5f19f51b898"}
Oct 30 01:27:17.161070 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:17.160991    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d113eb13e93afde9ac23f3e69dc517c868d7b777c1ea3575c091f5f19f51b898"
Oct 30 01:27:18.066868 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.066827    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main" containerID="cri-o://d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6" gracePeriod=45
Oct 30 01:27:18.328879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.328819    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 01:27:18.386357 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:27:18.386317    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:27:18.423144 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423107    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bed9833-4f1e-463d-a660-04bace8eb95b-kube-api-access-m7k64\" (UniqueName: \"kubernetes.io/projected/2bed9833-4f1e-463d-a660-04bace8eb95b-kube-api-access-m7k64\") pod \"2bed9833-4f1e-463d-a660-04bace8eb95b\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") "
Oct 30 01:27:18.423367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423216    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-home\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-home\") pod \"2bed9833-4f1e-463d-a660-04bace8eb95b\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") "
Oct 30 01:27:18.423451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423373    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-model-cache\") pod \"2bed9833-4f1e-463d-a660-04bace8eb95b\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") "
Oct 30 01:27:18.423451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423410    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-dshm\") pod \"2bed9833-4f1e-463d-a660-04bace8eb95b\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") "
Oct 30 01:27:18.423580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423502    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-home" pod "2bed9833-4f1e-463d-a660-04bace8eb95b" (UID: "2bed9833-4f1e-463d-a660-04bace8eb95b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:18.423580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423507    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-tmp-dir\") pod \"2bed9833-4f1e-463d-a660-04bace8eb95b\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") "
Oct 30 01:27:18.423580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423565    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bed9833-4f1e-463d-a660-04bace8eb95b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bed9833-4f1e-463d-a660-04bace8eb95b-tls-certs\") pod \"2bed9833-4f1e-463d-a660-04bace8eb95b\" (UID: \"2bed9833-4f1e-463d-a660-04bace8eb95b\") "
Oct 30 01:27:18.423740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423661    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-model-cache" pod "2bed9833-4f1e-463d-a660-04bace8eb95b" (UID: "2bed9833-4f1e-463d-a660-04bace8eb95b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:18.423740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423699    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-tmp-dir" pod "2bed9833-4f1e-463d-a660-04bace8eb95b" (UID: "2bed9833-4f1e-463d-a660-04bace8eb95b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:18.424006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.423983    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:18.424006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.424004    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:18.424006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.424013    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:18.425684 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.425662    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-dshm" pod "2bed9833-4f1e-463d-a660-04bace8eb95b" (UID: "2bed9833-4f1e-463d-a660-04bace8eb95b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:18.425777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.425740    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bed9833-4f1e-463d-a660-04bace8eb95b-kube-api-access-m7k64" pod "2bed9833-4f1e-463d-a660-04bace8eb95b" (UID: "2bed9833-4f1e-463d-a660-04bace8eb95b"). InnerVolumeSpecName "kube-api-access-m7k64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:18.426199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.426177    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bed9833-4f1e-463d-a660-04bace8eb95b-tls-certs" pod "2bed9833-4f1e-463d-a660-04bace8eb95b" (UID: "2bed9833-4f1e-463d-a660-04bace8eb95b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:18.525014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.524986    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bed9833-4f1e-463d-a660-04bace8eb95b-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:18.525014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.525012    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bed9833-4f1e-463d-a660-04bace8eb95b-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:18.525014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.525021    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7k64\" (UniqueName: \"kubernetes.io/projected/2bed9833-4f1e-463d-a660-04bace8eb95b-kube-api-access-m7k64\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:18.961525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:18.961480    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:19.175747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.175717    4576 generic.go:356] "Generic (PLEG): container finished" podID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerID="d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6" exitCode=0
Oct 30 01:27:19.176226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.175761    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"
Oct 30 01:27:19.176226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.175798    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" event={"ID":"2bed9833-4f1e-463d-a660-04bace8eb95b","Type":"ContainerDied","Data":"d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6"}
Oct 30 01:27:19.176226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.175842    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt" event={"ID":"2bed9833-4f1e-463d-a660-04bace8eb95b","Type":"ContainerDied","Data":"f54b24f6f3e73f28cbfd17966ff11f4f38dc1b58867709adedc470b30ab712c8"}
Oct 30 01:27:19.176226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.175859    4576 scope.go:122] "RemoveContainer" containerID="d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6"
Oct 30 01:27:19.185990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.185968    4576 scope.go:122] "RemoveContainer" containerID="d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6"
Oct 30 01:27:19.186350 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:27:19.186331    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6\": container with ID starting with d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6 not found: ID does not exist" containerID="d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6"
Oct 30 01:27:19.186428 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.186358    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6"} err="failed to get container status \"d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6\": rpc error: code = NotFound desc = could not find container \"d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6\": container with ID starting with d7df3652decf0d5fcc943b3a1ec443e30395ba2be7556570c5a4486c84b2bdc6 not found: ID does not exist"
Oct 30 01:27:19.232100 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.232030    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"]
Oct 30 01:27:19.245711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:19.245680    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8cf33a/simulator-w1-8cf33a-kserve-68998b644b-9c6bt"]
Oct 30 01:27:20.114930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:20.114875    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" path="/var/lib/kubelet/pods/2bed9833-4f1e-463d-a660-04bace8eb95b/volumes"
Oct 30 01:27:20.417043 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:20.416950    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:21.981497 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:21.981392    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:23.565536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:23.565502    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:24.800406 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:24.800361    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:24.966405 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:24.966369    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:25.081686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:25.081586    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:25.332610 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:25.332512    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:25.438351 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:25.438312    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:25.556528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:25.556494    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:25.976217 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:25.976183    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:27.035952 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:27.035912    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:28.159542 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:28.159502    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:27:28.305446 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:28.305409    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:28.931296 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:28.931226    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:29.070369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:29.070328    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"]
Oct 30 01:27:29.082423 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:29.082395    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8cf33a-qbb8w"]
Oct 30 01:27:29.106996 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:29.106962    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:29.233107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:29.233066    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:29.591675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:29.591567    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:30.106800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:30.106768    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bbd5b1-716f-4ae4-879d-643092107dc5" path="/var/lib/kubelet/pods/b9bbd5b1-716f-4ae4-879d-643092107dc5/volumes"
Oct 30 01:27:30.713974 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:30.713939    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:31.728080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:31.728041    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:31.851294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:31.851224    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:32.081893 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.081810    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:32.204333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.204298    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:32.578656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.578615    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"]
Oct 30 01:27:32.579115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579096    4576 cpu_manager.go:414] "containerMap: removing container" podUID="0e9e579c-4505-4f5a-b8cd-fe1182e77177" containerName="revoke-keys"
Oct 30 01:27:32.579260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579116    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9e579c-4505-4f5a-b8cd-fe1182e77177" containerName="revoke-keys"
Oct 30 01:27:32.579260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579133    4576 cpu_manager.go:414] "containerMap: removing container" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main"
Oct 30 01:27:32.579260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579141    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main"
Oct 30 01:27:32.579260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579167    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e32a2559-f34f-4062-aebf-360f1129da0a" containerName="revoke-keys"
Oct 30 01:27:32.579260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579175    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32a2559-f34f-4062-aebf-360f1129da0a" containerName="revoke-keys"
Oct 30 01:27:32.579616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579289    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32a2559-f34f-4062-aebf-360f1129da0a" containerName="revoke-keys"
Oct 30 01:27:32.579616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579312    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9e579c-4505-4f5a-b8cd-fe1182e77177" containerName="revoke-keys"
Oct 30 01:27:32.579616 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.579323    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bed9833-4f1e-463d-a660-04bace8eb95b" containerName="main"
Oct 30 01:27:32.583969 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.583949    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.603494 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.603470    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:27:32.603633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.603521    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:27:32.604306 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.604286    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"]
Oct 30 01:27:32.640812 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.640776    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:32.641082 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.640889    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:27:32.762924 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.762888    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.763426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.763034    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b82t\" (UniqueName: \"kubernetes.io/projected/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-kube-api-access-8b82t\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.763426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.763131    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.864693 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.864593    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.864693 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.864678    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.864951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.864782    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b82t\" (UniqueName: \"kubernetes.io/projected/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-kube-api-access-8b82t\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.865711 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.865572    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.867466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.867435    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.895265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.889910    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b82t\" (UniqueName: \"kubernetes.io/projected/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-kube-api-access-8b82t\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:32.901212 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:32.898456    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:33.204115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:33.204086    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"]
Oct 30 01:27:33.205694 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:27:33.205664    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8281cc6d_1fe4_4a4c_916a_15386f9fa9c4.slice/crio-e5f1246a758047c659f1430b0e920039faab94a765a42b341ebba1dfad0a9c55 WatchSource:0}: Error finding container e5f1246a758047c659f1430b0e920039faab94a765a42b341ebba1dfad0a9c55: Status 404 returned error can't find the container with id e5f1246a758047c659f1430b0e920039faab94a765a42b341ebba1dfad0a9c55
Oct 30 01:27:33.269632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:33.269593    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz" event={"ID":"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4","Type":"ContainerStarted","Data":"e5f1246a758047c659f1430b0e920039faab94a765a42b341ebba1dfad0a9c55"}
Oct 30 01:27:33.954991 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:33.954947    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:34.278345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:34.278249    4576 generic.go:356] "Generic (PLEG): container finished" podID="8281cc6d-1fe4-4a4c-916a-15386f9fa9c4" containerID="c03202091d9556543ab16a0ab8e80d02c96d87abc0d5f07569ec52824d37f337" exitCode=0
Oct 30 01:27:34.278345 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:34.278282    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz" event={"ID":"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4","Type":"ContainerDied","Data":"c03202091d9556543ab16a0ab8e80d02c96d87abc0d5f07569ec52824d37f337"}
Oct 30 01:27:35.267972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.267934    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:27:35.447910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.447879    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:35.581213 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.581128    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:35.594210 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.594184    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-kube-api-access-8b82t\" (UniqueName: \"kubernetes.io/projected/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-kube-api-access-8b82t\") pod \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") "
Oct 30 01:27:35.594372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.594307    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-xks-ca\") pod \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") "
Oct 30 01:27:35.594372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.594350    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-openshift-ca\") pod \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\" (UID: \"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4\") "
Oct 30 01:27:35.594730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.594703    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-openshift-ca" pod "8281cc6d-1fe4-4a4c-916a-15386f9fa9c4" (UID: "8281cc6d-1fe4-4a4c-916a-15386f9fa9c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:35.596505 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.596483    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-kube-api-access-8b82t" pod "8281cc6d-1fe4-4a4c-916a-15386f9fa9c4" (UID: "8281cc6d-1fe4-4a4c-916a-15386f9fa9c4"). InnerVolumeSpecName "kube-api-access-8b82t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:35.596583 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.596512    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-xks-ca" pod "8281cc6d-1fe4-4a4c-916a-15386f9fa9c4" (UID: "8281cc6d-1fe4-4a4c-916a-15386f9fa9c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:35.695833 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.695792    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b82t\" (UniqueName: \"kubernetes.io/projected/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-kube-api-access-8b82t\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:35.696014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.695847    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:35.696014 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:35.695864    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:27:36.079195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:36.079115    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:36.197686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:36.197639    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:36.300947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:36.300909    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"
Oct 30 01:27:36.301373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:36.300968    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz" event={"ID":"8281cc6d-1fe4-4a4c-916a-15386f9fa9c4","Type":"ContainerDied","Data":"e5f1246a758047c659f1430b0e920039faab94a765a42b341ebba1dfad0a9c55"}
Oct 30 01:27:36.301373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:36.301020    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5f1246a758047c659f1430b0e920039faab94a765a42b341ebba1dfad0a9c55"
Oct 30 01:27:36.397394 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:36.397320    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:37.368473 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:37.368433    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:38.880215 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:38.880144    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:39.883404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:39.883321    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:40.027769 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:40.027738    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:40.465664 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:40.465622    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:40.827352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:40.827265    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:41.911438 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:41.911404    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:42.652728 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:42.652698    4576 scope.go:122] "RemoveContainer" containerID="8fa473d12916ddb61b3202636a56f3956ee92f04b3d9b6b5a0c9bcfd6a472339"
Oct 30 01:27:42.662537 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:42.662512    4576 scope.go:122] "RemoveContainer" containerID="299c4209eaebd6113a7d20be097c40ab900d789889da0b7bcc2ef309e7952d37"
Oct 30 01:27:42.873099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:42.873038    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:43.200078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:43.200038    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:43.310946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:43.310905    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:43.558198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:43.558105    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:44.991253 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:44.991204    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:46.419978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:46.419930    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:46.812457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:46.812419    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:47.657056 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:47.657024    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:47.783282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:47.783251    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:48.239913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:48.239868    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:48.334660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:48.334615    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"]
Oct 30 01:27:48.347682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:48.347654    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-60f38f-8fvhz"]
Oct 30 01:27:48.368097 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:48.368065    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:48.382302 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:27:48.382257    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:27:48.476822 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:48.476793    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:48.592517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:48.592429    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:48.891023 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:48.890933    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:50.055634 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:50.055601    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:50.107571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:50.107536    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8281cc6d-1fe4-4a4c-916a-15386f9fa9c4" path="/var/lib/kubelet/pods/8281cc6d-1fe4-4a4c-916a-15386f9fa9c4/volumes"
Oct 30 01:27:50.748798 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:50.748766    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:27:51.651584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:51.651544    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:52.025489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:52.025456    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"]
Oct 30 01:27:52.039092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:52.039063    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f989bb2-djm28"]
Oct 30 01:27:52.107544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:52.107514    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057947eb-9d61-4733-852f-2b22e8bc9666" path="/var/lib/kubelet/pods/057947eb-9d61-4733-852f-2b22e8bc9666/volumes"
Oct 30 01:27:53.093801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:53.093763    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:54.506580 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.506547    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:54.626094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.626059    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:54.779073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.778985    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"]
Oct 30 01:27:54.779294 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.779194    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy" containerID="cri-o://b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281" gracePeriod=30
Oct 30 01:27:54.793889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.793860    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"]
Oct 30 01:27:54.794136 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.794112    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy" containerID="cri-o://363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0" gracePeriod=30
Oct 30 01:27:54.801069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.801037    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"]
Oct 30 01:27:54.801265 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.801213    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy" containerID="cri-o://b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f" gracePeriod=30
Oct 30 01:27:54.861343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:54.861308    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:56.441442 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:56.441389    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:57.977071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:57.977039    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:27:59.658629 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:27:59.658596    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:00.065484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.065448    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:28:00.107329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.107303    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:28:00.110912 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.110893    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:28:00.144582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144518    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-envoy\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.144582 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144569    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-envoy\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.144862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144604    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-data\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.144862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144635    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-token\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-token\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.144862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144663    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-data\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.144862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144686    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-credential-socket\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.144862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144755    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-kube-api-access-2njr6\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-kube-api-access-2njr6\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.144862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144783    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-credential-socket\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.144862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144813    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-podinfo\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.144862 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144854    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-socket\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144894    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-kube-api-access-dm2fl\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-kube-api-access-dm2fl\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144937    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-data\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.144967    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-credential-socket\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145007    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-token\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-token\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145037    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-token\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-token\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145066    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-certs\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145100    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-podinfo\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145133    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8944c80f-2076-4692-aaea-d95d0f23d00e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8944c80f-2076-4692-aaea-d95d0f23d00e-istiod-ca-cert\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145168    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6ef1e5-67c3-4889-bcd9-0e1260313283-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6ef1e5-67c3-4889-bcd9-0e1260313283-istiod-ca-cert\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145205    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istiod-ca-cert\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145255    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-certs\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145316    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-socket\") pod \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\" (UID: \"69b3c72c-700f-4ffb-af37-5998bf4a12e7\") "
Oct 30 01:28:00.145367 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145354    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-socket\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.146095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145384    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-podinfo\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.146095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145430    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-certs\") pod \"8944c80f-2076-4692-aaea-d95d0f23d00e\" (UID: \"8944c80f-2076-4692-aaea-d95d0f23d00e\") "
Oct 30 01:28:00.146095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145463    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-envoy\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.146095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.145511    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-kube-api-access-jtjpl\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-kube-api-access-jtjpl\") pod \"db6ef1e5-67c3-4889-bcd9-0e1260313283\" (UID: \"db6ef1e5-67c3-4889-bcd9-0e1260313283\") "
Oct 30 01:28:00.146733 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.146690    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-credential-socket" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.147458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.147431    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istiod-ca-cert" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:00.147604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.147582    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-socket" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.147700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.147598    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-certs" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.147961 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.147926    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-credential-socket" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.148670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.148185    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-credential-socket" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.148670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.148495    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-data" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.149130 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.149094    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-data" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.149458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.149312    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-socket" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.149458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.149196    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8944c80f-2076-4692-aaea-d95d0f23d00e-istiod-ca-cert" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:00.149631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.149478    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-certs" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.149631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.149493    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-data" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.149751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.149696    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-socket" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.149751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.149703    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6ef1e5-67c3-4889-bcd9-0e1260313283-istiod-ca-cert" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:00.150115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.150082    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-certs" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.154834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.154807    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-envoy" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.154955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.154837    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-envoy" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.154955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.154888    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-envoy" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:00.154955 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.154887    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-kube-api-access-2njr6" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "kube-api-access-2njr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.155083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.154999    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-podinfo" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:00.155121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.155093    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-token" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.155338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.155306    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-podinfo" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:00.155469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.155443    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-kube-api-access-jtjpl" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "kube-api-access-jtjpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.155555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.155509    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-token" pod "69b3c72c-700f-4ffb-af37-5998bf4a12e7" (UID: "69b3c72c-700f-4ffb-af37-5998bf4a12e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.155621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.155607    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-token" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.155794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.155771    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-podinfo" pod "db6ef1e5-67c3-4889-bcd9-0e1260313283" (UID: "db6ef1e5-67c3-4889-bcd9-0e1260313283"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:00.156379 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.156357    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-kube-api-access-dm2fl" pod "8944c80f-2076-4692-aaea-d95d0f23d00e" (UID: "8944c80f-2076-4692-aaea-d95d0f23d00e"). InnerVolumeSpecName "kube-api-access-dm2fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:00.246707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246672    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.246707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246703    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.246707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246715    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246728    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246739    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246750    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246760    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246771    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtjpl\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-kube-api-access-jtjpl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246783    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246795    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246805    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246815    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246825    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246836    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246847    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2njr6\" (UniqueName: \"kubernetes.io/projected/69b3c72c-700f-4ffb-af37-5998bf4a12e7-kube-api-access-2njr6\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246859    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6ef1e5-67c3-4889-bcd9-0e1260313283-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246870    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b3c72c-700f-4ffb-af37-5998bf4a12e7-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246881    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246894    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm2fl\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-kube-api-access-dm2fl\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246905    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246917    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8944c80f-2076-4692-aaea-d95d0f23d00e-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246929    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6ef1e5-67c3-4889-bcd9-0e1260313283-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246974    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.246989    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b3c72c-700f-4ffb-af37-5998bf4a12e7-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.247002    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8944c80f-2076-4692-aaea-d95d0f23d00e-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.247015    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8944c80f-2076-4692-aaea-d95d0f23d00e-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.247038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.247028    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6ef1e5-67c3-4889-bcd9-0e1260313283-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:00.467118 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.467081    4576 generic.go:356] "Generic (PLEG): container finished" podID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerID="363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0" exitCode=0
Oct 30 01:28:00.467347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.467129    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"
Oct 30 01:28:00.467347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.467163    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" event={"ID":"69b3c72c-700f-4ffb-af37-5998bf4a12e7","Type":"ContainerDied","Data":"363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0"}
Oct 30 01:28:00.467347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.467220    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn" event={"ID":"69b3c72c-700f-4ffb-af37-5998bf4a12e7","Type":"ContainerDied","Data":"56080fd7893d1386c09b3b6447ed1395b525a1406a4078ca2d723e97808bb128"}
Oct 30 01:28:00.467347 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.467265    4576 scope.go:122] "RemoveContainer" containerID="363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0"
Oct 30 01:28:00.468763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.468739    4576 generic.go:356] "Generic (PLEG): container finished" podID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerID="b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281" exitCode=0
Oct 30 01:28:00.468901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.468801    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"
Oct 30 01:28:00.468901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.468821    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" event={"ID":"8944c80f-2076-4692-aaea-d95d0f23d00e","Type":"ContainerDied","Data":"b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281"}
Oct 30 01:28:00.468901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.468873    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz" event={"ID":"8944c80f-2076-4692-aaea-d95d0f23d00e","Type":"ContainerDied","Data":"f80400f043bccbf470d2cd8c7d9e928cb390ba1789e1a4f2aeeef2a5b3c3eb89"}
Oct 30 01:28:00.470480 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.470456    4576 generic.go:356] "Generic (PLEG): container finished" podID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerID="b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f" exitCode=0
Oct 30 01:28:00.470593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.470485    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" event={"ID":"db6ef1e5-67c3-4889-bcd9-0e1260313283","Type":"ContainerDied","Data":"b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f"}
Oct 30 01:28:00.470593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.470506    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"
Oct 30 01:28:00.470593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.470534    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8" event={"ID":"db6ef1e5-67c3-4889-bcd9-0e1260313283","Type":"ContainerDied","Data":"10968b57089b154b1b5fd3a9baa91a417d973086f314860f43d1378ff746db42"}
Oct 30 01:28:00.478415 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.478398    4576 scope.go:122] "RemoveContainer" containerID="363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0"
Oct 30 01:28:00.478696 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:28:00.478675    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0\": container with ID starting with 363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0 not found: ID does not exist" containerID="363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0"
Oct 30 01:28:00.478772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.478706    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0"} err="failed to get container status \"363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0\": rpc error: code = NotFound desc = could not find container \"363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0\": container with ID starting with 363fc127f6a4dd51b85b7c9f68c604309fec0d6c356ba3cae635e5be1b9edbd0 not found: ID does not exist"
Oct 30 01:28:00.478772 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.478721    4576 scope.go:122] "RemoveContainer" containerID="b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281"
Oct 30 01:28:00.488699 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.488675    4576 scope.go:122] "RemoveContainer" containerID="b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281"
Oct 30 01:28:00.489004 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:28:00.488985    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281\": container with ID starting with b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281 not found: ID does not exist" containerID="b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281"
Oct 30 01:28:00.489077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.489013    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281"} err="failed to get container status \"b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281\": rpc error: code = NotFound desc = could not find container \"b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281\": container with ID starting with b56d5dfe4ecd9f1d69297e29b5be74f6c9b4ca755e261d33e4e328910f227281 not found: ID does not exist"
Oct 30 01:28:00.489077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.489030    4576 scope.go:122] "RemoveContainer" containerID="b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f"
Oct 30 01:28:00.498729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.498707    4576 scope.go:122] "RemoveContainer" containerID="b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f"
Oct 30 01:28:00.499076 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:28:00.499052    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f\": container with ID starting with b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f not found: ID does not exist" containerID="b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f"
Oct 30 01:28:00.499161 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.499086    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f"} err="failed to get container status \"b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f\": rpc error: code = NotFound desc = could not find container \"b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f\": container with ID starting with b85f12b6b4257617947037386bbe630b645f337a61bb100c2e3b17f17ed5163f not found: ID does not exist"
Oct 30 01:28:00.523101 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.523066    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"]
Oct 30 01:28:00.536474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.536443    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-r8rg8"]
Oct 30 01:28:00.573138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.573104    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"]
Oct 30 01:28:00.578448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.578420    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-229fn"]
Oct 30 01:28:00.615625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.615601    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"]
Oct 30 01:28:00.619020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:00.618995    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-60f38f-openshift-default-bc8f76b9c-mqmtz"]
Oct 30 01:28:01.111531 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:01.111439    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:02.025592 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:02.025551    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"]
Oct 30 01:28:02.039262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:02.039217    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1725b6af-6l6rs"]
Oct 30 01:28:02.106314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:02.106283    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" path="/var/lib/kubelet/pods/69b3c72c-700f-4ffb-af37-5998bf4a12e7/volumes"
Oct 30 01:28:02.106734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:02.106720    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" path="/var/lib/kubelet/pods/8944c80f-2076-4692-aaea-d95d0f23d00e/volumes"
Oct 30 01:28:02.107107 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:02.107095    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94cd3ef8-840a-4de0-ae7c-407eeaf59a3a" path="/var/lib/kubelet/pods/94cd3ef8-840a-4de0-ae7c-407eeaf59a3a/volumes"
Oct 30 01:28:02.107818 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:02.107804    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" path="/var/lib/kubelet/pods/db6ef1e5-67c3-4889-bcd9-0e1260313283/volumes"
Oct 30 01:28:02.708700 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:02.708667    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:04.163632 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:04.163596    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:05.628780 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:05.628742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:05.956350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:05.956311    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:06.072957 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:06.072909    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:06.860389 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:06.860327    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:07.443267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:07.443215    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:07.560946 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:07.560913    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:07.671680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:07.671648    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:08.063474 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:08.063444    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:09.194945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:09.194915    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:09.792150 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:09.792116    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:28:10.721343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:10.721305    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:12.370983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:12.370946    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:13.895834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:13.895800    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:14.004752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:14.004715    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:14.263950 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:14.263913    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"]
Oct 30 01:28:14.264181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:14.264146    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy" containerID="cri-o://46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508" gracePeriod=30
Oct 30 01:28:14.330555 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:14.330518    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:14.776910 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:14.776880    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:15.580529 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:15.580495    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:17.108709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:17.108666    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:18.238601 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:18.238558    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:18.381688 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:28:18.381647    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:28:18.566749 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:18.566652    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:19.513813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.513786    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:28:19.590222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.590142    4576 generic.go:356] "Generic (PLEG): container finished" podID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerID="46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508" exitCode=0
Oct 30 01:28:19.590398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.590216    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"
Oct 30 01:28:19.590398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.590260    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" event={"ID":"dde6c4b4-b01f-4636-8244-26a809e97b77","Type":"ContainerDied","Data":"46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508"}
Oct 30 01:28:19.590398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.590317    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz" event={"ID":"dde6c4b4-b01f-4636-8244-26a809e97b77","Type":"ContainerDied","Data":"d3edfcc8bb39ec3ff192a8317208692e3f9ada44156a04a92f31b27e27a4bcd3"}
Oct 30 01:28:19.590398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.590345    4576 scope.go:122] "RemoveContainer" containerID="46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508"
Oct 30 01:28:19.600465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.600449    4576 scope.go:122] "RemoveContainer" containerID="84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72"
Oct 30 01:28:19.610052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.610035    4576 scope.go:122] "RemoveContainer" containerID="46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508"
Oct 30 01:28:19.610324 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:28:19.610306    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508\": container with ID starting with 46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508 not found: ID does not exist" containerID="46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508"
Oct 30 01:28:19.610396 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.610334    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508"} err="failed to get container status \"46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508\": rpc error: code = NotFound desc = could not find container \"46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508\": container with ID starting with 46a06c9381ec13abd3b04278657a1daf91e64516e61eead117b282edfa23d508 not found: ID does not exist"
Oct 30 01:28:19.610396 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.610350    4576 scope.go:122] "RemoveContainer" containerID="84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72"
Oct 30 01:28:19.610589 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:28:19.610569    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72\": container with ID starting with 84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72 not found: ID does not exist" containerID="84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72"
Oct 30 01:28:19.610653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.610600    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72"} err="failed to get container status \"84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72\": rpc error: code = NotFound desc = could not find container \"84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72\": container with ID starting with 84b05a552c307e612d307e0e02a7b96219a79c4ba9454d9e7abfa60d5f4cac72 not found: ID does not exist"
Oct 30 01:28:19.641951 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.641921    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-token\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-token\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.641972    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde6c4b4-b01f-4636-8244-26a809e97b77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6c4b4-b01f-4636-8244-26a809e97b77-istiod-ca-cert\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642006    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-kube-api-access-kngb5\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-kube-api-access-kngb5\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642035    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-socket\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642122 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642069    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-envoy\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642355 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642246    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-certs\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642352    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-podinfo\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642418    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-credential-socket\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642432    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-socket" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:19.642489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642473    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-data\") pod \"dde6c4b4-b01f-4636-8244-26a809e97b77\" (UID: \"dde6c4b4-b01f-4636-8244-26a809e97b77\") "
Oct 30 01:28:19.642666 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642645    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-certs" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:19.642876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642824    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.642876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642849    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.643069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642892    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-data" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:19.643069 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.643040    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-credential-socket" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:19.643275 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.642958    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde6c4b4-b01f-4636-8244-26a809e97b77-istiod-ca-cert" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:19.644945 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.644911    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-token" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:19.645078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.644967    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-kube-api-access-kngb5" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "kube-api-access-kngb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:19.645151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.645088    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-envoy" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:19.645151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.645129    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-podinfo" pod "dde6c4b4-b01f-4636-8244-26a809e97b77" (UID: "dde6c4b4-b01f-4636-8244-26a809e97b77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:19.742754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.742719    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"]
Oct 30 01:28:19.743738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.743632    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.743738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.743662    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6c4b4-b01f-4636-8244-26a809e97b77-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.743738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.743679    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kngb5\" (UniqueName: \"kubernetes.io/projected/dde6c4b4-b01f-4636-8244-26a809e97b77-kube-api-access-kngb5\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.743738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.743690    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.743738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.743699    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.743738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.743711    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.743738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.743719    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6c4b4-b01f-4636-8244-26a809e97b77-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:19.848603 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.848494    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:19.940498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.940460    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"]
Oct 30 01:28:19.943628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:19.943602    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-954bf6-openshift-default-96d7fdb5c-hlddz"]
Oct 30 01:28:20.108168 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:20.108094    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" path="/var/lib/kubelet/pods/dde6c4b4-b01f-4636-8244-26a809e97b77/volumes"
Oct 30 01:28:21.261491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:21.261451    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:21.864907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:21.864874    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:22.027506 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:22.027468    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"]
Oct 30 01:28:22.041166 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:22.041132    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"]
Oct 30 01:28:22.042915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:22.042884    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba415567-nrltr"]
Oct 30 01:28:22.044391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:22.044372    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76de7d81-fh7v5"]
Oct 30 01:28:22.106282 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:22.106221    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e" path="/var/lib/kubelet/pods/1f2f3655-2cf9-4489-b5a8-7f33b5ce7a8e/volumes"
Oct 30 01:28:22.106683 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:22.106669    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b000e779-6421-4220-bd3e-5240b8b7bdea" path="/var/lib/kubelet/pods/b000e779-6421-4220-bd3e-5240b8b7bdea/volumes"
Oct 30 01:28:23.273686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:23.273635    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:24.025742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:24.025708    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"]
Oct 30 01:28:24.039426 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:24.039394    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43e98a65-qlzkt"]
Oct 30 01:28:24.106429 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:24.106385    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68371e2a-9cb2-48ae-bdc3-6d5e055ef91d" path="/var/lib/kubelet/pods/68371e2a-9cb2-48ae-bdc3-6d5e055ef91d/volumes"
Oct 30 01:28:24.713019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:24.712983    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:25.515169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:25.515137    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:25.626904 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:25.626869    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:26.251342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:26.251308    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:27.486386 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486351    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"]
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486714    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8281cc6d-1fe4-4a4c-916a-15386f9fa9c4" containerName="revoke-keys"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486729    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8281cc6d-1fe4-4a4c-916a-15386f9fa9c4" containerName="revoke-keys"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486748    4576 cpu_manager.go:414] "containerMap: removing container" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486754    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486765    4576 cpu_manager.go:414] "containerMap: removing container" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486770    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486784    4576 cpu_manager.go:414] "containerMap: removing container" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486788    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486796    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy"
Oct 30 01:28:27.486801 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486802    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy"
Oct 30 01:28:27.487189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486867    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b3c72c-700f-4ffb-af37-5998bf4a12e7" containerName="istio-proxy"
Oct 30 01:28:27.487189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486878    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6ef1e5-67c3-4889-bcd9-0e1260313283" containerName="istio-proxy"
Oct 30 01:28:27.487189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486888    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8944c80f-2076-4692-aaea-d95d0f23d00e" containerName="istio-proxy"
Oct 30 01:28:27.487189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486900    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy"
Oct 30 01:28:27.487189 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.486908    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8281cc6d-1fe4-4a4c-916a-15386f9fa9c4" containerName="revoke-keys"
Oct 30 01:28:27.491956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.491929    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.507025 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.506993    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"]
Oct 30 01:28:27.511564 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.511535    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b05856-7145-4175-9f79-116e325091cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.511745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.511581    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b05856-7145-4175-9f79-116e325091cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.511745 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.511610    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psgvt\" (UniqueName: \"kubernetes.io/projected/b9b05856-7145-4175-9f79-116e325091cf-kube-api-access-psgvt\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.553944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.553900    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:28:27.554373 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.554355    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:28:27.594922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.594892    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:28:27.594922 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.594913    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:28:27.612482 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.612446    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b05856-7145-4175-9f79-116e325091cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.612655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.612504    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b05856-7145-4175-9f79-116e325091cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.612655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.612542    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psgvt\" (UniqueName: \"kubernetes.io/projected/b9b05856-7145-4175-9f79-116e325091cf-kube-api-access-psgvt\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.613207 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.613145    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b05856-7145-4175-9f79-116e325091cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.615068 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.615045    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b05856-7145-4175-9f79-116e325091cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.633751 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.633721    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psgvt\" (UniqueName: \"kubernetes.io/projected/b9b05856-7145-4175-9f79-116e325091cf-kube-api-access-psgvt\") pod \"maas-api-revoke-sub-6fc699e1-vpxwq\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:27.808222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:27.808133    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:28.107110 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:28.107085    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"]
Oct 30 01:28:28.108182 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:28:28.108162    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b05856_7145_4175_9f79_116e325091cf.slice/crio-224a8e53336a38c78548d0933254cb602c97327c5987f33a770638ca3f1cc2a8 WatchSource:0}: Error finding container 224a8e53336a38c78548d0933254cb602c97327c5987f33a770638ca3f1cc2a8: Status 404 returned error can't find the container with id 224a8e53336a38c78548d0933254cb602c97327c5987f33a770638ca3f1cc2a8
Oct 30 01:28:28.656799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:28.656689    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq" event={"ID":"b9b05856-7145-4175-9f79-116e325091cf","Type":"ContainerStarted","Data":"4d0d50d83273362574d4c99e08c60a8f81a21582908933a8840ef9296ecb5f4c"}
Oct 30 01:28:28.657312 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:28.656807    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq" event={"ID":"b9b05856-7145-4175-9f79-116e325091cf","Type":"ContainerStarted","Data":"224a8e53336a38c78548d0933254cb602c97327c5987f33a770638ca3f1cc2a8"}
Oct 30 01:28:28.701124 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:28.701072    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq" podStartSLOduration=1.701057258 podStartE2EDuration="1.701057258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:28.699453164 +0000 UTC m=+2988.770777000" watchObservedRunningTime="2026-10-30 01:28:28.701057258 +0000 UTC m=+2988.772381132"
Oct 30 01:28:29.664613 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:29.664576    4576 generic.go:356] "Generic (PLEG): container finished" podID="b9b05856-7145-4175-9f79-116e325091cf" containerID="4d0d50d83273362574d4c99e08c60a8f81a21582908933a8840ef9296ecb5f4c" exitCode=0
Oct 30 01:28:29.665062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:29.664663    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq" event={"ID":"b9b05856-7145-4175-9f79-116e325091cf","Type":"ContainerDied","Data":"4d0d50d83273362574d4c99e08c60a8f81a21582908933a8840ef9296ecb5f4c"}
Oct 30 01:28:30.809633 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.809610    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:30.840488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.840451    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b05856-7145-4175-9f79-116e325091cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b05856-7145-4175-9f79-116e325091cf-maas-api-xks-ca\") pod \"b9b05856-7145-4175-9f79-116e325091cf\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") "
Oct 30 01:28:30.840656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.840540    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b05856-7145-4175-9f79-116e325091cf-kube-api-access-psgvt\" (UniqueName: \"kubernetes.io/projected/b9b05856-7145-4175-9f79-116e325091cf-kube-api-access-psgvt\") pod \"b9b05856-7145-4175-9f79-116e325091cf\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") "
Oct 30 01:28:30.840656 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.840640    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b05856-7145-4175-9f79-116e325091cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b05856-7145-4175-9f79-116e325091cf-maas-api-openshift-ca\") pod \"b9b05856-7145-4175-9f79-116e325091cf\" (UID: \"b9b05856-7145-4175-9f79-116e325091cf\") "
Oct 30 01:28:30.840979 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.840955    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b05856-7145-4175-9f79-116e325091cf-maas-api-openshift-ca" pod "b9b05856-7145-4175-9f79-116e325091cf" (UID: "b9b05856-7145-4175-9f79-116e325091cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:30.842879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.842855    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b05856-7145-4175-9f79-116e325091cf-maas-api-xks-ca" pod "b9b05856-7145-4175-9f79-116e325091cf" (UID: "b9b05856-7145-4175-9f79-116e325091cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:30.842956 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.842872    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b05856-7145-4175-9f79-116e325091cf-kube-api-access-psgvt" pod "b9b05856-7145-4175-9f79-116e325091cf" (UID: "b9b05856-7145-4175-9f79-116e325091cf"). InnerVolumeSpecName "kube-api-access-psgvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:30.942098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.942062    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b05856-7145-4175-9f79-116e325091cf-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:30.942098 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.942097    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psgvt\" (UniqueName: \"kubernetes.io/projected/b9b05856-7145-4175-9f79-116e325091cf-kube-api-access-psgvt\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:30.942365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:30.942114    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b05856-7145-4175-9f79-116e325091cf-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:31.678430 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:31.678399    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"
Oct 30 01:28:31.678631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:31.678428    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq" event={"ID":"b9b05856-7145-4175-9f79-116e325091cf","Type":"ContainerDied","Data":"224a8e53336a38c78548d0933254cb602c97327c5987f33a770638ca3f1cc2a8"}
Oct 30 01:28:31.678631 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:31.678485    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="224a8e53336a38c78548d0933254cb602c97327c5987f33a770638ca3f1cc2a8"
Oct 30 01:28:34.025730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:34.025692    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"]
Oct 30 01:28:34.039789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:34.039752    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475c543a-tzj8h"]
Oct 30 01:28:34.107138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:34.107095    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05c2802-6d7e-4e37-b594-87476c9a7083" path="/var/lib/kubelet/pods/b05c2802-6d7e-4e37-b594-87476c9a7083/volumes"
Oct 30 01:28:34.765518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:34.765477    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main" containerID="cri-o://a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e" gracePeriod=45
Oct 30 01:28:35.014747 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.014721    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 01:28:35.077197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.077110    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tmp-dir\") pod \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") "
Oct 30 01:28:35.077197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.077159    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-home\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-home\") pod \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") "
Oct 30 01:28:35.077937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.077227    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-model-cache\") pod \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") "
Oct 30 01:28:35.077937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.077317    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tls-certs\") pod \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") "
Oct 30 01:28:35.077937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.077359    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-dshm\") pod \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") "
Oct 30 01:28:35.077937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.077409    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-kube-api-access-lt6n4\" (UniqueName: \"kubernetes.io/projected/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-kube-api-access-lt6n4\") pod \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\" (UID: \"ed86ab45-a7e9-4335-b98a-8a98dd88f83e\") "
Oct 30 01:28:35.077937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.077509    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tmp-dir" pod "ed86ab45-a7e9-4335-b98a-8a98dd88f83e" (UID: "ed86ab45-a7e9-4335-b98a-8a98dd88f83e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:35.077937 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.077811    4576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tmp-dir\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:35.078326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.078070    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-model-cache" pod "ed86ab45-a7e9-4335-b98a-8a98dd88f83e" (UID: "ed86ab45-a7e9-4335-b98a-8a98dd88f83e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:35.078326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.078299    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-home" pod "ed86ab45-a7e9-4335-b98a-8a98dd88f83e" (UID: "ed86ab45-a7e9-4335-b98a-8a98dd88f83e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:35.080223 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.080196    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-kube-api-access-lt6n4" pod "ed86ab45-a7e9-4335-b98a-8a98dd88f83e" (UID: "ed86ab45-a7e9-4335-b98a-8a98dd88f83e"). InnerVolumeSpecName "kube-api-access-lt6n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:35.080653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.080634    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-dshm" pod "ed86ab45-a7e9-4335-b98a-8a98dd88f83e" (UID: "ed86ab45-a7e9-4335-b98a-8a98dd88f83e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:35.080734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.080655    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tls-certs" pod "ed86ab45-a7e9-4335-b98a-8a98dd88f83e" (UID: "ed86ab45-a7e9-4335-b98a-8a98dd88f83e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:35.178333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.178301    4576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-home\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:35.178333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.178330    4576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-model-cache\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:35.178333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.178340    4576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-tls-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:35.178565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.178348    4576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-dshm\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:35.178565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.178357    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt6n4\" (UniqueName: \"kubernetes.io/projected/ed86ab45-a7e9-4335-b98a-8a98dd88f83e-kube-api-access-lt6n4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:35.702146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.702106    4576 generic.go:356] "Generic (PLEG): container finished" podID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerID="a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e" exitCode=0
Oct 30 01:28:35.702357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.702165    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"
Oct 30 01:28:35.702357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.702194    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" event={"ID":"ed86ab45-a7e9-4335-b98a-8a98dd88f83e","Type":"ContainerDied","Data":"a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e"}
Oct 30 01:28:35.702357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.702264    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl" event={"ID":"ed86ab45-a7e9-4335-b98a-8a98dd88f83e","Type":"ContainerDied","Data":"c71da735fbaf2068804c5e8edd730e32f78aac9ba003254e7cd1a66f9816693d"}
Oct 30 01:28:35.702357 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.702289    4576 scope.go:122] "RemoveContainer" containerID="a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e"
Oct 30 01:28:35.712465 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.712443    4576 scope.go:122] "RemoveContainer" containerID="a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e"
Oct 30 01:28:35.712759 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:28:35.712739    4576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e\": container with ID starting with a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e not found: ID does not exist" containerID="a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e"
Oct 30 01:28:35.712829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.712773    4576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e"} err="failed to get container status \"a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e\": rpc error: code = NotFound desc = could not find container \"a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e\": container with ID starting with a6499920b6a5018fcf9dd2801d4e509d5b3c377f667985c9034b649550aed16e not found: ID does not exist"
Oct 30 01:28:35.758114 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.758084    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"]
Oct 30 01:28:35.772061 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:35.772018    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b565ec/simulator-w4-b565ec-kserve-d68c79f-w8czl"]
Oct 30 01:28:36.106536 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:36.106457    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" path="/var/lib/kubelet/pods/ed86ab45-a7e9-4335-b98a-8a98dd88f83e/volumes"
Oct 30 01:28:40.189165 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:40.189127    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:28:42.652518 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:42.652479    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:42.700829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:42.700789    4576 scope.go:122] "RemoveContainer" containerID="9374b1fc07ef1625081ad0bfab170d38ee5d9145ec667ba2ce9d4802152931f0"
Oct 30 01:28:42.710552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:42.710532    4576 scope.go:122] "RemoveContainer" containerID="c67c657d75d82332215910ff7ebe14678dd46ded20f5816bf7e712bfb75ca72f"
Oct 30 01:28:42.720007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:42.719985    4576 scope.go:122] "RemoveContainer" containerID="43ab6524556fa702245f24ce08d745efdbb08694a22c391be287e38c0ec596a5"
Oct 30 01:28:42.986914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:42.986877    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:43.108428 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:43.108393    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:43.214750 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:43.214715    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:43.677797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:43.677755    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:46.574608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:46.574565    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:28:47.775915 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:47.775862    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:48.381042 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:28:48.381005    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:28:48.592894 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:48.592864    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:49.331214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:49.331174    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:49.445077 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:49.445044    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:49.584301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:49.584185    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"]
Oct 30 01:28:49.584491 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:49.584453    4576 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy" containerID="cri-o://11ab48e2a5466fc0097284856a5216c50c3f0284db3619372d9d994df4b77eda" gracePeriod=30
Oct 30 01:28:49.660561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:49.660509    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:53.291833 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:53.291752    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:54.825889 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.825857    4576 generic.go:356] "Generic (PLEG): container finished" podID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerID="11ab48e2a5466fc0097284856a5216c50c3f0284db3619372d9d994df4b77eda" exitCode=0
Oct 30 01:28:54.826321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.825932    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" event={"ID":"6aebd8f6-9479-449c-9cb4-364cd0737de0","Type":"ContainerDied","Data":"11ab48e2a5466fc0097284856a5216c50c3f0284db3619372d9d994df4b77eda"}
Oct 30 01:28:54.861378 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.861296    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.962708    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-data\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.962786    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-token\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-token\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.962825    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-certs\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.962864    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-socket\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.962925    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-kube-api-access-5cvct\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-kube-api-access-5cvct\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.962962    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-podinfo\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.962975    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-data" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.962995    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-envoy\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.963037    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aebd8f6-9479-449c-9cb4-364cd0737de0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aebd8f6-9479-449c-9cb4-364cd0737de0-istiod-ca-cert\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.963097    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-credential-socket\") pod \"6aebd8f6-9479-449c-9cb4-364cd0737de0\" (UID: \"6aebd8f6-9479-449c-9cb4-364cd0737de0\") "
Oct 30 01:28:54.963261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.963156    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-socket" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:54.964090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.963434    4576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:54.964090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.963459    4576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-data\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:54.964090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.963652    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-credential-socket" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:54.964090 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.963670    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-certs" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:54.964369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.964167    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aebd8f6-9479-449c-9cb4-364cd0737de0-istiod-ca-cert" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:54.965719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.965682    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-token" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:54.966342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.966314    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-envoy" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:28:54.966448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.966367    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-podinfo" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:28:54.966637 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:54.966623    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-kube-api-access-5cvct" pod "6aebd8f6-9479-449c-9cb4-364cd0737de0" (UID: "6aebd8f6-9479-449c-9cb4-364cd0737de0"). InnerVolumeSpecName "kube-api-access-5cvct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:55.064284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.064248    4576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-token\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:55.064284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.064276    4576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-workload-certs\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:55.064284 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.064285    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvct\" (UniqueName: \"kubernetes.io/projected/6aebd8f6-9479-449c-9cb4-364cd0737de0-kube-api-access-5cvct\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:55.064549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.064297    4576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-podinfo\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:55.064549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.064307    4576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-istio-envoy\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:55.064549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.064314    4576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aebd8f6-9479-449c-9cb4-364cd0737de0-istiod-ca-cert\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:55.064549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.064322    4576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aebd8f6-9479-449c-9cb4-364cd0737de0-credential-socket\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:28:55.846514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.845012    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"
Oct 30 01:28:55.846514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.845051    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l" event={"ID":"6aebd8f6-9479-449c-9cb4-364cd0737de0","Type":"ContainerDied","Data":"5463670448a1e8742717fff5ec6ddb9eabb915c2adc20451e84d57c830e47b8f"}
Oct 30 01:28:55.846514 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.845121    4576 scope.go:122] "RemoveContainer" containerID="11ab48e2a5466fc0097284856a5216c50c3f0284db3619372d9d994df4b77eda"
Oct 30 01:28:55.933377 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.933259    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:55.933869 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.933810    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"]
Oct 30 01:28:55.940382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:55.940356    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b565ec-openshift-default-785c49d9bf-zl49l"]
Oct 30 01:28:56.042553 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:56.042508    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:28:56.075789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:56.075397    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mb4zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:28:56.075789 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:56.075486    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mb4zz" podUID="4ee221b6-8c75-451b-b949-e90edd2185a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:28:56.123030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:56.122947    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" path="/var/lib/kubelet/pods/6aebd8f6-9479-449c-9cb4-364cd0737de0/volumes"
Oct 30 01:28:57.491375 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:28:57.491323    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:02.965278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:02.965216    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:09.143689 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:09.143644    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:18.380967 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:29:18.380925    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:29:27.294732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:27.294683    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:29.243953 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:29.243913    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:35.355660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.355628    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:35.788729 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.788692    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"]
Oct 30 01:29:35.789073 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789060    4576 cpu_manager.go:414] "containerMap: removing container" podUID="b9b05856-7145-4175-9f79-116e325091cf" containerName="revoke-keys"
Oct 30 01:29:35.789119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789074    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b05856-7145-4175-9f79-116e325091cf" containerName="revoke-keys"
Oct 30 01:29:35.789119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789087    4576 cpu_manager.go:414] "containerMap: removing container" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy"
Oct 30 01:29:35.789119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789092    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy"
Oct 30 01:29:35.789119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789104    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main"
Oct 30 01:29:35.789119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789109    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main"
Oct 30 01:29:35.789119 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789120    4576 cpu_manager.go:414] "containerMap: removing container" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy"
Oct 30 01:29:35.789338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789124    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy"
Oct 30 01:29:35.789338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789178    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aebd8f6-9479-449c-9cb4-364cd0737de0" containerName="istio-proxy"
Oct 30 01:29:35.789338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789187    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde6c4b4-b01f-4636-8244-26a809e97b77" containerName="istio-proxy"
Oct 30 01:29:35.789338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789195    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b05856-7145-4175-9f79-116e325091cf" containerName="revoke-keys"
Oct 30 01:29:35.789338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.789203    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed86ab45-a7e9-4335-b98a-8a98dd88f83e" containerName="main"
Oct 30 01:29:35.792754 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.792734    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.827326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.827293    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"]
Oct 30 01:29:35.835334 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835308    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.835444 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835347    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.835517 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835449    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.835561 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835550    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.835630 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835576    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.835691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835659    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.835758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835703    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.835820 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835778    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87t2\" (UniqueName: \"kubernetes.io/projected/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-kube-api-access-d87t2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.835820 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.835810    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.936884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.936848    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.936899    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937027    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937108 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937082    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937121    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937159    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87t2\" (UniqueName: \"kubernetes.io/projected/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-kube-api-access-d87t2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937189    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937306    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937507 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937342    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937563    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.937817 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937770    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.938009 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.937984    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.938191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.938167    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.938368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.938346    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.939484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.939460    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.939797 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.939778    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.958182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.958149    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:35.958458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:35.958433    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87t2\" (UniqueName: \"kubernetes.io/projected/deb10f38-0a8d-428e-a7bf-8aa9d4392baf-kube-api-access-d87t2\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bbcwh\" (UID: \"deb10f38-0a8d-428e-a7bf-8aa9d4392baf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:36.105866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:36.105792    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:36.416782 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:36.416668    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"]
Oct 30 01:29:36.418946 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:29:36.418921    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb10f38_0a8d_428e_a7bf_8aa9d4392baf.slice/crio-ed2ae6ce631c1271a27b6854571755de5042547de72f69b3a75c1fe0235a595b WatchSource:0}: Error finding container ed2ae6ce631c1271a27b6854571755de5042547de72f69b3a75c1fe0235a595b: Status 404 returned error can't find the container with id ed2ae6ce631c1271a27b6854571755de5042547de72f69b3a75c1fe0235a595b
Oct 30 01:29:36.421030 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:36.421000    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:29:36.421093 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:36.421072    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:29:36.421133 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:36.421117    4576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 01:29:37.099065 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:37.099022    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" event={"ID":"deb10f38-0a8d-428e-a7bf-8aa9d4392baf","Type":"ContainerStarted","Data":"7f5bce5348cdb7e9a3c28f780bf3ad26cbf1e0d63d4fad0ce07323a4e2e19231"}
Oct 30 01:29:37.099283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:37.099115    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" event={"ID":"deb10f38-0a8d-428e-a7bf-8aa9d4392baf","Type":"ContainerStarted","Data":"ed2ae6ce631c1271a27b6854571755de5042547de72f69b3a75c1fe0235a595b"}
Oct 30 01:29:37.106535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:37.106512    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:37.107824 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:37.107799    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bbcwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 01:29:37.107972 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:37.107860    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podUID="deb10f38-0a8d-428e-a7bf-8aa9d4392baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 01:29:37.163725 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:37.163671    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podStartSLOduration=2.163658728 podStartE2EDuration="2.163658728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:37.158073698 +0000 UTC m=+3057.229397535" watchObservedRunningTime="2026-10-30 01:29:37.163658728 +0000 UTC m=+3057.234982563"
Oct 30 01:29:38.106311 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:38.106274    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bbcwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 01:29:38.106746 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:38.106332    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podUID="deb10f38-0a8d-428e-a7bf-8aa9d4392baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 01:29:39.107022 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:39.106984    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bbcwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 01:29:39.107565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:39.107048    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podUID="deb10f38-0a8d-428e-a7bf-8aa9d4392baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 01:29:40.106286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:40.106257    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bbcwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 01:29:40.106463 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:40.106311    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podUID="deb10f38-0a8d-428e-a7bf-8aa9d4392baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 01:29:41.106608 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:41.106568    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bbcwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 01:29:41.107019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:41.106625    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podUID="deb10f38-0a8d-428e-a7bf-8aa9d4392baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 01:29:42.106627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.106594    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bbcwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 01:29:42.107157 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.106648    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podUID="deb10f38-0a8d-428e-a7bf-8aa9d4392baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 01:29:42.778662 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.778622    4576 scope.go:122] "RemoveContainer" containerID="edb4c42663cbab195578374139295bf2ee31ac0259445c1916391be4a7efe4c4"
Oct 30 01:29:42.788607 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.788586    4576 scope.go:122] "RemoveContainer" containerID="fc84190ca28c785f408c6f7741de61cc39d526578f169cb5f9f4d5085ecefb23"
Oct 30 01:29:42.797787 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.797767    4576 scope.go:122] "RemoveContainer" containerID="1aa5eb21504b108e6bf65be91768f19914d36c23af8da74a7e4398117519815e"
Oct 30 01:29:42.807336 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.807310    4576 scope.go:122] "RemoveContainer" containerID="656a4e5aaf728fff00a7562c7e54d58d276f3cbd085341d70a569e9eb825f53a"
Oct 30 01:29:42.820866 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.820837    4576 scope.go:122] "RemoveContainer" containerID="c1c72234702f0df3a7b4435d9fd618be8210882b2c5dabf99b472dd0c0e6a218"
Oct 30 01:29:42.830690 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.830658    4576 scope.go:122] "RemoveContainer" containerID="fb4090cfdabb7f67f442ce4f4c17e67b92e079704327c418366fa236e2bce6b3"
Oct 30 01:29:42.842329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.842304    4576 scope.go:122] "RemoveContainer" containerID="0ee960fed50b799b96b9d070ea1db267c7ef8c1a900348b56ac61126f0a5a381"
Oct 30 01:29:42.851911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.851886    4576 scope.go:122] "RemoveContainer" containerID="069df0c83e1d2d0ccb4bcf0021efde98717e6557171cc1c42770354054cabba6"
Oct 30 01:29:42.863183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.863159    4576 scope.go:122] "RemoveContainer" containerID="1419364690e6c194a5f33a8d57b3ec4d9f072ddaf785ca206b5405292cf9cbb5"
Oct 30 01:29:42.872433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.872413    4576 scope.go:122] "RemoveContainer" containerID="4600f8d172cfa2bb23d9a9335df392c1b379dbd59a586846554d881bb170e9e1"
Oct 30 01:29:42.881421 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:42.881385    4576 scope.go:122] "RemoveContainer" containerID="78b1e96e90880fa732206dec2826a97848079a240acb12e4ffe8db412804708c"
Oct 30 01:29:43.106884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:43.106796    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bbcwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 01:29:43.106884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:43.106851    4576 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podUID="deb10f38-0a8d-428e-a7bf-8aa9d4392baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 01:29:43.109588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:43.109541    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:44.677026 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:44.676989    4576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:44.679680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:44.677351    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:44.679680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:44.678145    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh"
Oct 30 01:29:47.887842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:47.887802    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:48.379850 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:29:48.379808    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:29:49.724484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:49.724448    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:52.790160 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:52.790120    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:52.905587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:52.905552    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:29:58.373604 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:29:58.373570    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:30:00.730409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:00.730367    4576 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bbcwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:30:00.731117 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:00.731083    4576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bbcwh" podUID="deb10f38-0a8d-428e-a7bf-8aa9d4392baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:30:01.036279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:01.036156    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"]
Oct 30 01:30:01.049401 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:01.049358    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"]
Oct 30 01:30:01.052058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:01.052029    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d036e807-llzwl"]
Oct 30 01:30:01.054468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:01.054441    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d25ff4-24tc4"]
Oct 30 01:30:02.107331 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:02.107298    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153e78da-54d6-4f12-8aff-d7d0bb9e57cd" path="/var/lib/kubelet/pods/153e78da-54d6-4f12-8aff-d7d0bb9e57cd/volumes"
Oct 30 01:30:02.107732 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:02.107701    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effc843b-dc8e-4e6b-a64e-46b11a2e51e7" path="/var/lib/kubelet/pods/effc843b-dc8e-4e6b-a64e-46b11a2e51e7/volumes"
Oct 30 01:30:04.399968 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.399930    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"]
Oct 30 01:30:04.404675 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.404656    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.421903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.421876    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"]
Oct 30 01:30:04.434066 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.434035    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:04.434222 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.434093    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:30:04.435707 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.435687    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:30:04.435854 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.435731    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:30:04.445432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.445385    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.445590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.445441    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgf4\" (UniqueName: \"kubernetes.io/projected/8bb12190-79f7-4ad2-922b-68c65902743b-kube-api-access-vmgf4\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.445590 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.445506    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.545930 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.545900    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.546140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.545943    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgf4\" (UniqueName: \"kubernetes.io/projected/8bb12190-79f7-4ad2-922b-68c65902743b-kube-api-access-vmgf4\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.546140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.546065    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.546593 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.546571    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.548643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.548622    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.566903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.566874    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgf4\" (UniqueName: \"kubernetes.io/projected/8bb12190-79f7-4ad2-922b-68c65902743b-kube-api-access-vmgf4\") pod \"maas-api-revoke-sub-c1e271a9-9p566\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:04.717169 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:04.717133    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:05.012455 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:05.012425    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"]
Oct 30 01:30:05.014376 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:30:05.014348    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb12190_79f7_4ad2_922b_68c65902743b.slice/crio-6ffbf10009919c0ba4dcdee4dc96db09730b1506d2ab525fd99cdab7ad5730ad WatchSource:0}: Error finding container 6ffbf10009919c0ba4dcdee4dc96db09730b1506d2ab525fd99cdab7ad5730ad: Status 404 returned error can't find the container with id 6ffbf10009919c0ba4dcdee4dc96db09730b1506d2ab525fd99cdab7ad5730ad
Oct 30 01:30:05.279303 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:05.279272    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566" event={"ID":"8bb12190-79f7-4ad2-922b-68c65902743b","Type":"ContainerStarted","Data":"6ffbf10009919c0ba4dcdee4dc96db09730b1506d2ab525fd99cdab7ad5730ad"}
Oct 30 01:30:06.286321 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:06.286284    4576 generic.go:356] "Generic (PLEG): container finished" podID="8bb12190-79f7-4ad2-922b-68c65902743b" containerID="00918a33a2c2ab4204c96a2654f260ff765c88b65e6b2f02393ed94e995987b3" exitCode=0
Oct 30 01:30:06.286734 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:06.286353    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566" event={"ID":"8bb12190-79f7-4ad2-922b-68c65902743b","Type":"ContainerDied","Data":"00918a33a2c2ab4204c96a2654f260ff765c88b65e6b2f02393ed94e995987b3"}
Oct 30 01:30:07.424227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.424194    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:07.474816 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.474779    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-openshift-ca\") pod \"8bb12190-79f7-4ad2-922b-68c65902743b\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") "
Oct 30 01:30:07.474973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.474840    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb12190-79f7-4ad2-922b-68c65902743b-kube-api-access-vmgf4\" (UniqueName: \"kubernetes.io/projected/8bb12190-79f7-4ad2-922b-68c65902743b-kube-api-access-vmgf4\") pod \"8bb12190-79f7-4ad2-922b-68c65902743b\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") "
Oct 30 01:30:07.474973 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.474898    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-xks-ca\") pod \"8bb12190-79f7-4ad2-922b-68c65902743b\" (UID: \"8bb12190-79f7-4ad2-922b-68c65902743b\") "
Oct 30 01:30:07.475151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.475123    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-openshift-ca" pod "8bb12190-79f7-4ad2-922b-68c65902743b" (UID: "8bb12190-79f7-4ad2-922b-68c65902743b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:07.475335 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.475313    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:30:07.477205 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.477176    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-xks-ca" pod "8bb12190-79f7-4ad2-922b-68c65902743b" (UID: "8bb12190-79f7-4ad2-922b-68c65902743b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:07.477344 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.477263    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb12190-79f7-4ad2-922b-68c65902743b-kube-api-access-vmgf4" pod "8bb12190-79f7-4ad2-922b-68c65902743b" (UID: "8bb12190-79f7-4ad2-922b-68c65902743b"). InnerVolumeSpecName "kube-api-access-vmgf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:07.576719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.576632    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmgf4\" (UniqueName: \"kubernetes.io/projected/8bb12190-79f7-4ad2-922b-68c65902743b-kube-api-access-vmgf4\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:30:07.576719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:07.576664    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb12190-79f7-4ad2-922b-68c65902743b-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:30:08.305434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:08.305400    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"
Oct 30 01:30:08.305653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:08.305428    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566" event={"ID":"8bb12190-79f7-4ad2-922b-68c65902743b","Type":"ContainerDied","Data":"6ffbf10009919c0ba4dcdee4dc96db09730b1506d2ab525fd99cdab7ad5730ad"}
Oct 30 01:30:08.305653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:08.305501    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ffbf10009919c0ba4dcdee4dc96db09730b1506d2ab525fd99cdab7ad5730ad"
Oct 30 01:30:15.561226 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:15.561180    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:30:18.381786 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:30:18.381750    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:30:20.468794 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:20.468764    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:30:22.612313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:22.611725    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:30:22.742258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:22.741686    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:30:23.468571 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:23.468530    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:30:31.026410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:31.026372    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"]
Oct 30 01:30:31.040589 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:31.040544    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"]
Oct 30 01:30:31.042182 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:31.042150    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc681c33-ptkkh"]
Oct 30 01:30:31.043587 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:31.043562    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789b2252-b9mh9"]
Oct 30 01:30:32.106761 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:32.106725    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02270e01-1b63-40eb-ad59-84e8efe6bec4" path="/var/lib/kubelet/pods/02270e01-1b63-40eb-ad59-84e8efe6bec4/volumes"
Oct 30 01:30:32.107159 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:32.107107    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3831674-af14-4ae2-9721-c3055b8a9e68" path="/var/lib/kubelet/pods/e3831674-af14-4ae2-9721-c3055b8a9e68/volumes"
Oct 30 01:30:33.025646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:33.025613    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"]
Oct 30 01:30:33.039532 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:33.039501    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66eca326-k5sds"]
Oct 30 01:30:34.106227 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:34.106191    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741e0095-af6d-4322-8b11-b158a24e2946" path="/var/lib/kubelet/pods/741e0095-af6d-4322-8b11-b158a24e2946/volumes"
Oct 30 01:30:36.026641 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:36.026601    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"]
Oct 30 01:30:36.040591 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:36.040559    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"]
Oct 30 01:30:36.041903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:36.041884    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8652274e-j6mgt"]
Oct 30 01:30:36.043239 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:36.043219    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fe78c1-g8f8p"]
Oct 30 01:30:36.106911 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:36.106875    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0e9a78-d9a4-499e-835a-4e20ed85db0c" path="/var/lib/kubelet/pods/8b0e9a78-d9a4-499e-835a-4e20ed85db0c/volumes"
Oct 30 01:30:36.107301 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:36.107286    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e094331b-78e5-451d-80e4-92da6142ed29" path="/var/lib/kubelet/pods/e094331b-78e5-451d-80e4-92da6142ed29/volumes"
Oct 30 01:30:38.779840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:38.779800    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:30:42.959469 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:42.959433    4576 scope.go:122] "RemoveContainer" containerID="a1b3ce2902a2b0ff7f813367bccda659d18523f395f2899cb71b52b78cdf5104"
Oct 30 01:30:42.988454 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:42.987871    4576 scope.go:122] "RemoveContainer" containerID="fd11c949e4d49a1a6b1ec18d16c768b3dc77e2da61221c0cfe32df8b57d70e14"
Oct 30 01:30:43.042375 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:43.042351    4576 scope.go:122] "RemoveContainer" containerID="69382a9f189457e6742f0ce1911feb44a319237d5bcbc81cf8d2cf6b885d74c4"
Oct 30 01:30:43.057647 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:43.057622    4576 scope.go:122] "RemoveContainer" containerID="18d427254fe1adf2592333e7704db3d06dcda0482cd8315b7909353f1797dbf6"
Oct 30 01:30:43.065913 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:43.065892    4576 scope.go:122] "RemoveContainer" containerID="4207a2466fa7abe60f372a15c26309bd555c5dec256042c4d29efb455b4b7b88"
Oct 30 01:30:43.141356 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:43.141327    4576 scope.go:122] "RemoveContainer" containerID="6878fec1aa7b37040782da01309aa041cc22159fe18041282d3c4ba014154208"
Oct 30 01:30:43.153058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:43.153039    4576 scope.go:122] "RemoveContainer" containerID="79df3450ea745b6341c3d049528103edae6aff9e14c4391ab52e19bfbc5978f6"
Oct 30 01:30:43.364926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:43.364887    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:30:48.379779 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:30:48.379739    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:30:49.837596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:49.837540    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:30:50.030579 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:50.030538    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:30:54.071290 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:30:54.071251    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:31:17.660138 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:17.660095    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:31:18.383631 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:31:18.383590    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:31:27.845535 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:27.845497    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:31:31.986248 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:31.986199    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:31:43.344639 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.344604    4576 scope.go:122] "RemoveContainer" containerID="a41fe65947ccaafd576fcc8c1f08ff0644d2ba70d7961d4fd0c9642414fc5e65"
Oct 30 01:31:43.353876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.353854    4576 scope.go:122] "RemoveContainer" containerID="425a124368d5221c237d376f2f7ac4e0a3b4e7def1765081edb00f59f3b040e8"
Oct 30 01:31:43.362111 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.362090    4576 scope.go:122] "RemoveContainer" containerID="8e41c1afa8f8aef49e1becf6c733f477110d47f5301ba5150d16ab2d6fd1d2d6"
Oct 30 01:31:43.370276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.370255    4576 scope.go:122] "RemoveContainer" containerID="2a4a1bb2c4424dc7ea120231400701522de7a3e81709e86569919ee2f2ad4bcf"
Oct 30 01:31:43.378921 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.378902    4576 scope.go:122] "RemoveContainer" containerID="896a5d5f2aeba3541ae1e1703318ff43c68b559d6eb19fbc3ebb212045fe57c2"
Oct 30 01:31:43.387316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.387291    4576 scope.go:122] "RemoveContainer" containerID="3d6c465aa2bec992291e39016ff3d093ec692ecae6d2f918a12b223890c69477"
Oct 30 01:31:43.395487 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.395467    4576 scope.go:122] "RemoveContainer" containerID="12bc850d4e946405642a7c9d8fd4468a92930e3241464de3883bb7c34804b2da"
Oct 30 01:31:43.403280 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.403262    4576 scope.go:122] "RemoveContainer" containerID="f4c8429da209c48830a428e6a30342754d693c7f4fd524255cfa6f8920821f49"
Oct 30 01:31:43.411799 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.411774    4576 scope.go:122] "RemoveContainer" containerID="8198f0e88bb1df3c004140dfee8ef75c41d94f2031628210e71d93aa2d97d649"
Oct 30 01:31:43.420409 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.420388    4576 scope.go:122] "RemoveContainer" containerID="da32b6401dacbb6a0ea1033601436dd47f3e2e2a77476376a1b6fb4d8ca2a99e"
Oct 30 01:31:43.428433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.428412    4576 scope.go:122] "RemoveContainer" containerID="9b7138a2de42c2a3a8bb7f0d72fc31f725341377e2ef46c3e88632599982bce4"
Oct 30 01:31:43.436477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.436454    4576 scope.go:122] "RemoveContainer" containerID="74b3620fea5a4cd39bb86f473c2ea883b1e2271d45f7db4400f22469d1a5b8ef"
Oct 30 01:31:43.444262 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.444225    4576 scope.go:122] "RemoveContainer" containerID="a1e15beb17bc2f80a1383d8578e5c0466e1666a3454b3cef1c5d5d66ab190a2e"
Oct 30 01:31:43.461501 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:43.461477    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:31:48.381023 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:31:48.380985    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:31:49.261742 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:49.261709    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:31:49.974193 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:49.974151    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:31:50.152696 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:50.152664    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:31:54.364565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:31:54.364475    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:32:10.756938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:10.756905    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:32:14.982370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:14.982335    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:32:17.038305 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:17.038263    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"]
Oct 30 01:32:17.052511 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:17.052470    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"]
Oct 30 01:32:17.053917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:17.053886    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cee14b6-dn4kw"]
Oct 30 01:32:17.055948 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:17.055919    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ceedf74-2tcpg"]
Oct 30 01:32:18.108792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:18.108754    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9e579c-4505-4f5a-b8cd-fe1182e77177" path="/var/lib/kubelet/pods/0e9e579c-4505-4f5a-b8cd-fe1182e77177/volumes"
Oct 30 01:32:18.109355 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:18.109201    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32a2559-f34f-4062-aebf-360f1129da0a" path="/var/lib/kubelet/pods/e32a2559-f34f-4062-aebf-360f1129da0a/volumes"
Oct 30 01:32:18.380222 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:32:18.380141    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:32:31.464944 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:31.464906    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:32:43.507292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:43.507256    4576 scope.go:122] "RemoveContainer" containerID="f146b5b5f999992464c527717f1c4b88381ebd794134e25446b9c2caa4c73c67"
Oct 30 01:32:43.515826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:43.515803    4576 scope.go:122] "RemoveContainer" containerID="7dbaa26ed321951b38f0790b77eed1e4fab8f46467d0a1c663ef676dc47b49b9"
Oct 30 01:32:43.670731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:43.670702    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:32:48.381582 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:32:48.381540    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:32:48.402874 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:48.402842    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:32:50.092626 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:50.092591    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:32:50.263540 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:50.263501    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:32:59.090279 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:32:59.090220    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:33:04.380468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:04.380429    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:33:18.380072 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:33:18.380037    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:33:21.057489 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:21.057451    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:33:25.800180 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:25.800143    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:33:31.028172 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:31.028122    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"]
Oct 30 01:33:31.042052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:31.042008    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fc699e1-vpxwq"]
Oct 30 01:33:32.107039 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:32.106995    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b05856-7145-4175-9f79-116e325091cf" path="/var/lib/kubelet/pods/b9b05856-7145-4175-9f79-116e325091cf/volumes"
Oct 30 01:33:43.539773 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:43.539739    4576 scope.go:122] "RemoveContainer" containerID="c03202091d9556543ab16a0ab8e80d02c96d87abc0d5f07569ec52824d37f337"
Oct 30 01:33:43.548488 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:43.548465    4576 scope.go:122] "RemoveContainer" containerID="16f48f72cebded730018d40e0e9e833fee5f835c9b82aa50aaff2d6b9b3b43c9"
Oct 30 01:33:43.556622 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:43.556601    4576 scope.go:122] "RemoveContainer" containerID="38a86b1095b31a2098509db25341ad541ea551a8669e2b3fe3083fa1c160ecaf"
Oct 30 01:33:43.564402 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:43.564376    4576 scope.go:122] "RemoveContainer" containerID="8397b45dd815199e25eeec2a7d1362f90393c1aed5d44a5640904c397c8c7b67"
Oct 30 01:33:43.571779 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:43.571755    4576 scope.go:122] "RemoveContainer" containerID="2c7b596b49dd202a8a316269d27d1d78408f4af074e9641813739ee481d4e488"
Oct 30 01:33:44.644686 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:44.644655    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:44.685387 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:44.685353    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:33:48.380269 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:33:48.380214    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:33:49.569199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:49.569173    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:33:50.177338 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:50.177305    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:54.155544 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:54.155507    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:54.597208 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:54.597164    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:54.832504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:54.832469    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:58.169914 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:58.169883    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:58.305006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:58.304954    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:59.729140 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:59.729107    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:59.861777 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:59.861747    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:33:59.975130 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:33:59.975090    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:01.453916 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:01.453882    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:02.924556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:02.924526    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:03.202203 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:03.202165    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:34:04.806670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:04.806617    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:06.373398 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:06.373354    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:08.001595 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:08.001562    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:09.647149 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:09.647118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:11.192370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:11.192331    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:12.987054 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:12.987020    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:13.103721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:13.103684    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:13.747963 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:13.747924    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:13.870330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:13.870272    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:14.581585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:14.581549    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:15.155605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:15.155576    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:16.100261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:16.100211    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:17.401197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:17.401162    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:34:17.817197 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:17.817160    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:18.380893 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:34:18.380860    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:34:19.384349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:19.384318    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:21.070770 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:21.070730    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:22.528266 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:22.528218    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:24.065460 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:24.065419    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:25.555276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:25.555209    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:27.078585 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:27.078541    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:28.597115 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:28.597081    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:30.144558 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:30.144526    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:31.663619 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:31.663587    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:32.642121 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:32.642085    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:34:33.137725 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:33.137685    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:34.682411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:34.682377    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:36.214990 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:36.214957    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:37.653840 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:37.653797    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:39.184883 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:39.184843    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:40.867528 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:40.867484    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:42.306466 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:42.306429    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:43.602422 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:43.602393    4576 scope.go:122] "RemoveContainer" containerID="4d0d50d83273362574d4c99e08c60a8f81a21582908933a8840ef9296ecb5f4c"
Oct 30 01:34:43.806823 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:43.806788    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:43.924983 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:43.924894    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:45.194549 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:45.194517    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:45.368658 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:45.368625    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:46.822586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:46.822546    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:48.255058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:48.255025    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:48.380248 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:34:48.380197    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:34:49.514586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:49.514554    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:49.817813 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:49.817723    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:50.009766 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:50.009737    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:34:50.432283 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:50.432227    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:51.576341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:51.576256    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:53.021038 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:53.021006    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:54.548330 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:54.548290    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:54.679380 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:54.679325    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:56.105644 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:56.105609    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:57.626800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:57.626755    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:34:59.170826 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:34:59.170786    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:00.259126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:00.259092    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:00.373492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:00.373455    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:00.697365 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:00.697332    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:01.968448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:01.968412    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:02.446058 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:02.446016    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:02.582605 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:02.582572    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:03.094126 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:03.094093    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:04.153362 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:04.153325    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:35:08.025752 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:08.025715    4576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"]
Oct 30 01:35:08.039432 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:08.039395    4576 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9p566"]
Oct 30 01:35:08.106034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:08.105994    4576 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb12190-79f7-4ad2-922b-68c65902743b" path="/var/lib/kubelet/pods/8bb12190-79f7-4ad2-922b-68c65902743b/volumes"
Oct 30 01:35:08.764384 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:08.764345    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:08.890502 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:08.890466    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:09.127576 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:09.127492    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:18.380612 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:35:18.380573    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:35:22.164860 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:22.164826    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:35:27.670214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:27.670172    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:35:29.663019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:29.662968    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:31.223199 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:31.223159    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:31.970045 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:31.970006    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:37.465400 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:37.465348    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:43.920758 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:43.920713    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:35:48.380681 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:35:48.380635    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:35:50.444131 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:35:50.444094    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:36:11.448850 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:36:11.448790    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:36:11.559776 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:36:11.559742    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:36:18.381333 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:36:18.381294    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:36:18.979247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:36:18.979196    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:36:43.628315 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:36:43.628280    4576 scope.go:122] "RemoveContainer" containerID="00918a33a2c2ab4204c96a2654f260ff765c88b65e6b2f02393ed94e995987b3"
Oct 30 01:36:44.041864 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:36:44.041807    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:36:48.380818 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:36:48.380771    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:36:50.553261 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:36:50.553214    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:36:55.166247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:36:55.166200    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:37:02.145574 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:02.145530    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:02.274250 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:02.274201    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:10.172543 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:10.172498    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:37:18.380318 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:37:18.380282    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:37:27.210427 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:27.210382    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:27.707267 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:27.707199    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:27.829005 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:27.828946    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:31.563175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:31.563142    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:31.677147 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:31.677118    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:32.970702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:32.970674    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:33.176917 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:33.176886    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:33.371710 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:33.371617    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:33.504589 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:33.504547    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:34.784994 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:34.784965    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:36.280481 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:36.280449    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:36.511584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:36.511551    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:37:37.920981 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:37.920940    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:39.560884 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:39.560852    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:41.163612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:41.163581    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:42.872096 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:42.872059    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:44.137586 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:44.137553    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:44.364370 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:44.364333    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:46.076971 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:46.076918    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:46.474078 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:46.474045    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:47.060504 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:47.060465    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:47.192943 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:47.192911    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:47.326697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:47.326621    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:47.873573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:47.873539    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:48.108697 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:48.108656    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:48.385126 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:37:48.385093    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:37:49.450831 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:49.450794    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:49.500390 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:49.500354    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:37:50.678343 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:50.678310    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:51.049247 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:51.049201    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:52.760092 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:52.760060    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:54.283627 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:54.283596    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:55.813350 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:55.813297    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:57.360625 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:57.360588    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:37:58.916556 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:37:58.916525    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:00.465274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:00.465218    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:02.182368 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:02.182336    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:03.660083 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:03.660045    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:05.375876 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:05.375833    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:05.962691 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:05.962653    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:38:06.858341 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:06.858303    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:08.478846 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:08.478813    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:09.940220 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:09.940187    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:11.387878 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:11.387843    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:12.977897 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:12.977869    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:14.495059 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:14.495025    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:15.961020 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:15.960986    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:17.523046 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:17.523010    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:18.380513 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:38:18.380480    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:38:18.900636 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:18.900598    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:19.117317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:19.117280    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:19.474411 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:19.474377    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:38:20.937167 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:20.937129    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:22.461288 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:22.461251    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:23.939731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:23.939695    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:24.052095 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:24.052061    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:25.466903 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:25.466864    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:27.174621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:27.174584    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:28.604484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:28.604444    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:29.652569 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:29.652529    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:29.769906 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:29.769872    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:29.892064 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:29.892020    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:31.006031 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:31.005997    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:31.112198 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:31.112168    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:31.616947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:31.616906    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:32.943989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:32.943950    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:38:37.641132 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:37.641073    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:37.756962 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:37.756916    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:37.990682 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:37.990637    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:42.311764 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:42.311716    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:44.192156 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:44.192115    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:38:44.262434 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:44.262398    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:48.381551 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:38:48.381516    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:38:50.783458 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:50.783417    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:56.440274 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:56.440210    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:38:56.885827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:38:56.885730    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:39:02.601521 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:02.601477    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:39:04.505129 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:04.505086    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:39:06.558300 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:06.558265    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:39:18.380936 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:39:18.380896    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:39:21.777931 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:21.777847    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:39:31.881646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:31.881609    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:39:33.668395 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:33.668359    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:39:34.618418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:34.618378    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:39:40.804462 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:40.804427    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:39:44.370919 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:44.370870    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:39:48.380398 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:39:48.380361    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:39:49.363649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:49.363613    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:39:50.984258 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:50.982262    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:39:51.233456 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:51.233419    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:39:56.548034 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:39:56.547995    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:40:12.875006 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:12.874972    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:40:18.380567 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:40:18.380523    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:40:22.272057 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:22.272017    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:40:30.494565 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:30.494524    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:40:37.326554 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.326519    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"]
Oct 30 01:40:37.326964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.326911    4576 cpu_manager.go:414] "containerMap: removing container" podUID="8bb12190-79f7-4ad2-922b-68c65902743b" containerName="revoke-keys"
Oct 30 01:40:37.326964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.326922    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb12190-79f7-4ad2-922b-68c65902743b" containerName="revoke-keys"
Oct 30 01:40:37.327045 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.326987    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb12190-79f7-4ad2-922b-68c65902743b" containerName="revoke-keys"
Oct 30 01:40:37.330146 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.330129    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.352534 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.352509    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"]
Oct 30 01:40:37.364333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.364310    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:40:37.364333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.364314    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:40:37.364333 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.364340    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:40:37.364568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.364392    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:40:37.365680 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.365652    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9bz\" (UniqueName: \"kubernetes.io/projected/ffd79150-1053-4599-9d34-b7af829a2937-kube-api-access-jz9bz\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.365804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.365713    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.365804 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.365791    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.466676 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.466641    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9bz\" (UniqueName: \"kubernetes.io/projected/ffd79150-1053-4599-9d34-b7af829a2937-kube-api-access-jz9bz\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.466880 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.466721    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.466880 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.466784    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.467551 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.467479    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.469352 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.469328    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.487299 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.487273    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9bz\" (UniqueName: \"kubernetes.io/projected/ffd79150-1053-4599-9d34-b7af829a2937-kube-api-access-jz9bz\") pod \"maas-api-revoke-sub-f03629db-rkzgr\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.642492 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.642410    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:37.934960 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:37.934932    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"]
Oct 30 01:40:37.937782 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:40:37.937743    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd79150_1053_4599_9d34_b7af829a2937.slice/crio-1fe8460876bf0177c7496577be0b5cbd40796469a74750b900835968e1e8c065 WatchSource:0}: Error finding container 1fe8460876bf0177c7496577be0b5cbd40796469a74750b900835968e1e8c065: Status 404 returned error can't find the container with id 1fe8460876bf0177c7496577be0b5cbd40796469a74750b900835968e1e8c065
Oct 30 01:40:38.463313 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:38.463272    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr" event={"ID":"ffd79150-1053-4599-9d34-b7af829a2937","Type":"ContainerStarted","Data":"3647e95ed56779712c80b0a69d64ff4900df337e4406a5c8cb14de8646d2de7b"}
Oct 30 01:40:38.463738 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:38.463350    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr" event={"ID":"ffd79150-1053-4599-9d34-b7af829a2937","Type":"ContainerStarted","Data":"1fe8460876bf0177c7496577be0b5cbd40796469a74750b900835968e1e8c065"}
Oct 30 01:40:38.519091 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:38.519039    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr" podStartSLOduration=1.5190243479999999 podStartE2EDuration="1.519024348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:38.504782114 +0000 UTC m=+3718.576105950" watchObservedRunningTime="2026-10-30 01:40:38.519024348 +0000 UTC m=+3718.590348182"
Oct 30 01:40:39.469842 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:39.469801    4576 generic.go:356] "Generic (PLEG): container finished" podID="ffd79150-1053-4599-9d34-b7af829a2937" containerID="3647e95ed56779712c80b0a69d64ff4900df337e4406a5c8cb14de8646d2de7b" exitCode=0
Oct 30 01:40:39.470287 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:39.469892    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr" event={"ID":"ffd79150-1053-4599-9d34-b7af829a2937","Type":"ContainerDied","Data":"3647e95ed56779712c80b0a69d64ff4900df337e4406a5c8cb14de8646d2de7b"}
Oct 30 01:40:40.609485 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.609458    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:40.696490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.696463    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd79150-1053-4599-9d34-b7af829a2937-kube-api-access-jz9bz\" (UniqueName: \"kubernetes.io/projected/ffd79150-1053-4599-9d34-b7af829a2937-kube-api-access-jz9bz\") pod \"ffd79150-1053-4599-9d34-b7af829a2937\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") "
Oct 30 01:40:40.696670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.696505    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-openshift-ca\") pod \"ffd79150-1053-4599-9d34-b7af829a2937\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") "
Oct 30 01:40:40.696670 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.696546    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-xks-ca\") pod \"ffd79150-1053-4599-9d34-b7af829a2937\" (UID: \"ffd79150-1053-4599-9d34-b7af829a2937\") "
Oct 30 01:40:40.696879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.696856    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-openshift-ca" pod "ffd79150-1053-4599-9d34-b7af829a2937" (UID: "ffd79150-1053-4599-9d34-b7af829a2937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:40:40.698809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.698781    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd79150-1053-4599-9d34-b7af829a2937-kube-api-access-jz9bz" pod "ffd79150-1053-4599-9d34-b7af829a2937" (UID: "ffd79150-1053-4599-9d34-b7af829a2937"). InnerVolumeSpecName "kube-api-access-jz9bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:40:40.698926 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.698814    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-xks-ca" pod "ffd79150-1053-4599-9d34-b7af829a2937" (UID: "ffd79150-1053-4599-9d34-b7af829a2937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:40:40.798099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.798020    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:40:40.798099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.798053    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz9bz\" (UniqueName: \"kubernetes.io/projected/ffd79150-1053-4599-9d34-b7af829a2937-kube-api-access-jz9bz\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:40:40.798099 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:40.798063    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd79150-1053-4599-9d34-b7af829a2937-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:40:41.479964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:41.479922    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr" event={"ID":"ffd79150-1053-4599-9d34-b7af829a2937","Type":"ContainerDied","Data":"1fe8460876bf0177c7496577be0b5cbd40796469a74750b900835968e1e8c065"}
Oct 30 01:40:41.479964 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:41.479948    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rkzgr"
Oct 30 01:40:41.480191 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:41.479980    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fe8460876bf0177c7496577be0b5cbd40796469a74750b900835968e1e8c065"
Oct 30 01:40:43.968727 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:43.968688    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:40:44.511660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:44.511627    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:40:46.660036 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:46.659996    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:40:48.450111 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:40:48.450057    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:40:49.751736 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.751696    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"]
Oct 30 01:40:49.752328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.752066    4576 cpu_manager.go:414] "containerMap: removing container" podUID="ffd79150-1053-4599-9d34-b7af829a2937" containerName="revoke-keys"
Oct 30 01:40:49.752328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.752076    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd79150-1053-4599-9d34-b7af829a2937" containerName="revoke-keys"
Oct 30 01:40:49.752328 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.752133    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd79150-1053-4599-9d34-b7af829a2937" containerName="revoke-keys"
Oct 30 01:40:49.755151 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.755120    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.774271 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.774220    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"]
Oct 30 01:40:49.785597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.785557    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:40:49.785597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.785572    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:40:49.785597 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.785596    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:40:49.785843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.785671    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:40:49.832053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.831938    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.832053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.832076    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.832053 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.832107    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwv5z\" (UniqueName: \"kubernetes.io/projected/33e689d7-6d8e-4433-8b0a-36fab7df563c-kube-api-access-nwv5z\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.933183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.933146    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.933183 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.933191    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwv5z\" (UniqueName: \"kubernetes.io/projected/33e689d7-6d8e-4433-8b0a-36fab7df563c-kube-api-access-nwv5z\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.933457 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.933227    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.933938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.933912    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.935901 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.935877    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:49.954477 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:49.954443    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwv5z\" (UniqueName: \"kubernetes.io/projected/33e689d7-6d8e-4433-8b0a-36fab7df563c-kube-api-access-nwv5z\") pod \"maas-api-revoke-sub-0165d7d2-nt2wm\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:50.068371 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:50.068285    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:50.363940 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:50.363724    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"]
Oct 30 01:40:50.366914 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:40:50.366885    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e689d7_6d8e_4433_8b0a_36fab7df563c.slice/crio-c0dae2360644f2befd0a4d0bb7d59077df7dcfd056888b7dcc9780355735b06b WatchSource:0}: Error finding container c0dae2360644f2befd0a4d0bb7d59077df7dcfd056888b7dcc9780355735b06b: Status 404 returned error can't find the container with id c0dae2360644f2befd0a4d0bb7d59077df7dcfd056888b7dcc9780355735b06b
Oct 30 01:40:50.529278 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:50.529242    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm" event={"ID":"33e689d7-6d8e-4433-8b0a-36fab7df563c","Type":"ContainerStarted","Data":"c0dae2360644f2befd0a4d0bb7d59077df7dcfd056888b7dcc9780355735b06b"}
Oct 30 01:40:51.102484 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:51.102453    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:40:51.359989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:51.359907    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:40:51.535342 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:51.535304    4576 generic.go:356] "Generic (PLEG): container finished" podID="33e689d7-6d8e-4433-8b0a-36fab7df563c" containerID="4c76bbf57258f82a9a24d3bf846fdac9db6f8dcf460fe7dbf0533adbf81598aa" exitCode=0
Oct 30 01:40:51.535527 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:51.535367    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm" event={"ID":"33e689d7-6d8e-4433-8b0a-36fab7df563c","Type":"ContainerDied","Data":"4c76bbf57258f82a9a24d3bf846fdac9db6f8dcf460fe7dbf0533adbf81598aa"}
Oct 30 01:40:52.677071 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.677044    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:52.766260 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.766202    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-openshift-ca\") pod \"33e689d7-6d8e-4433-8b0a-36fab7df563c\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") "
Oct 30 01:40:52.766468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.766328    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e689d7-6d8e-4433-8b0a-36fab7df563c-kube-api-access-nwv5z\" (UniqueName: \"kubernetes.io/projected/33e689d7-6d8e-4433-8b0a-36fab7df563c-kube-api-access-nwv5z\") pod \"33e689d7-6d8e-4433-8b0a-36fab7df563c\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") "
Oct 30 01:40:52.766468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.766385    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-xks-ca\") pod \"33e689d7-6d8e-4433-8b0a-36fab7df563c\" (UID: \"33e689d7-6d8e-4433-8b0a-36fab7df563c\") "
Oct 30 01:40:52.766596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.766572    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-openshift-ca" pod "33e689d7-6d8e-4433-8b0a-36fab7df563c" (UID: "33e689d7-6d8e-4433-8b0a-36fab7df563c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:40:52.766673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.766654    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:40:52.768621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.768597    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-xks-ca" pod "33e689d7-6d8e-4433-8b0a-36fab7df563c" (UID: "33e689d7-6d8e-4433-8b0a-36fab7df563c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:40:52.768721 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.768672    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e689d7-6d8e-4433-8b0a-36fab7df563c-kube-api-access-nwv5z" pod "33e689d7-6d8e-4433-8b0a-36fab7df563c" (UID: "33e689d7-6d8e-4433-8b0a-36fab7df563c"). InnerVolumeSpecName "kube-api-access-nwv5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:40:52.867448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.867364    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwv5z\" (UniqueName: \"kubernetes.io/projected/33e689d7-6d8e-4433-8b0a-36fab7df563c-kube-api-access-nwv5z\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:40:52.867448 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:52.867395    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e689d7-6d8e-4433-8b0a-36fab7df563c-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:40:53.545329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:53.545297    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm"
Oct 30 01:40:53.545329 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:53.545308    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-nt2wm" event={"ID":"33e689d7-6d8e-4433-8b0a-36fab7df563c","Type":"ContainerDied","Data":"c0dae2360644f2befd0a4d0bb7d59077df7dcfd056888b7dcc9780355735b06b"}
Oct 30 01:40:53.545584 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:40:53.545370    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0dae2360644f2befd0a4d0bb7d59077df7dcfd056888b7dcc9780355735b06b"
Oct 30 01:41:00.641643 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.641610    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"]
Oct 30 01:41:00.642075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.641952    4576 cpu_manager.go:414] "containerMap: removing container" podUID="33e689d7-6d8e-4433-8b0a-36fab7df563c" containerName="revoke-keys"
Oct 30 01:41:00.642075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.641964    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e689d7-6d8e-4433-8b0a-36fab7df563c" containerName="revoke-keys"
Oct 30 01:41:00.642075 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.642056    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e689d7-6d8e-4433-8b0a-36fab7df563c" containerName="revoke-keys"
Oct 30 01:41:00.645372 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.645346    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.663326 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.663296    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"]
Oct 30 01:41:00.675175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.675145    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:41:00.675175 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.675145    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:41:00.675418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.675206    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:41:00.675418 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.675206    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94jff\""
Oct 30 01:41:00.840164 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.840124    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.840349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.840208    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.840349 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.840324    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk96l\" (UniqueName: \"kubernetes.io/projected/e6fb3a61-756c-47af-ac43-7a078a22b19c-kube-api-access-sk96l\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.941052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.941002    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.941297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.941082    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.941297 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.941100    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk96l\" (UniqueName: \"kubernetes.io/projected/e6fb3a61-756c-47af-ac43-7a078a22b19c-kube-api-access-sk96l\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.941829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.941806    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.943655 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.943628    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:00.962525 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:00.962494    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk96l\" (UniqueName: \"kubernetes.io/projected/e6fb3a61-756c-47af-ac43-7a078a22b19c-kube-api-access-sk96l\") pod \"maas-api-revoke-sub-6b19338f-6ns7c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:01.258570 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:01.258485    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:01.555355 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:01.555327    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"]
Oct 30 01:41:01.557824 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:41:01.557800    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fb3a61_756c_47af_ac43_7a078a22b19c.slice/crio-fef3a9ce3c91e2c9e2b8154bfc0078d423846187a19eed64cf5be511ae3e7ba0 WatchSource:0}: Error finding container fef3a9ce3c91e2c9e2b8154bfc0078d423846187a19eed64cf5be511ae3e7ba0: Status 404 returned error can't find the container with id fef3a9ce3c91e2c9e2b8154bfc0078d423846187a19eed64cf5be511ae3e7ba0
Oct 30 01:41:01.579433 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:01.579405    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c" event={"ID":"e6fb3a61-756c-47af-ac43-7a078a22b19c","Type":"ContainerStarted","Data":"fef3a9ce3c91e2c9e2b8154bfc0078d423846187a19eed64cf5be511ae3e7ba0"}
Oct 30 01:41:02.585214 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:02.585177    4576 generic.go:356] "Generic (PLEG): container finished" podID="e6fb3a61-756c-47af-ac43-7a078a22b19c" containerID="c56b21dcdff9b995e276d180f703e4b30f21216c08bd7b7b6384457b93e0fb21" exitCode=0
Oct 30 01:41:02.585660 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:02.585263    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c" event={"ID":"e6fb3a61-756c-47af-ac43-7a078a22b19c","Type":"ContainerDied","Data":"c56b21dcdff9b995e276d180f703e4b30f21216c08bd7b7b6384457b93e0fb21"}
Oct 30 01:41:03.726019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.725991    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:03.868361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.868264    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-xks-ca\") pod \"e6fb3a61-756c-47af-ac43-7a078a22b19c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") "
Oct 30 01:41:03.868361 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.868314    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-openshift-ca\") pod \"e6fb3a61-756c-47af-ac43-7a078a22b19c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") "
Oct 30 01:41:03.868621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.868401    4576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fb3a61-756c-47af-ac43-7a078a22b19c-kube-api-access-sk96l\" (UniqueName: \"kubernetes.io/projected/e6fb3a61-756c-47af-ac43-7a078a22b19c-kube-api-access-sk96l\") pod \"e6fb3a61-756c-47af-ac43-7a078a22b19c\" (UID: \"e6fb3a61-756c-47af-ac43-7a078a22b19c\") "
Oct 30 01:41:03.868724 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.868699    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-openshift-ca" pod "e6fb3a61-756c-47af-ac43-7a078a22b19c" (UID: "e6fb3a61-756c-47af-ac43-7a078a22b19c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:41:03.870702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.870674    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-xks-ca" pod "e6fb3a61-756c-47af-ac43-7a078a22b19c" (UID: "e6fb3a61-756c-47af-ac43-7a078a22b19c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:41:03.870702 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.870684    4576 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fb3a61-756c-47af-ac43-7a078a22b19c-kube-api-access-sk96l" pod "e6fb3a61-756c-47af-ac43-7a078a22b19c" (UID: "e6fb3a61-756c-47af-ac43-7a078a22b19c"). InnerVolumeSpecName "kube-api-access-sk96l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:03.969498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.969459    4576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk96l\" (UniqueName: \"kubernetes.io/projected/e6fb3a61-756c-47af-ac43-7a078a22b19c-kube-api-access-sk96l\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:41:03.969498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.969489    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-xks-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:41:03.969498 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:03.969503    4576 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb3a61-756c-47af-ac43-7a078a22b19c-maas-api-openshift-ca\") on node \"ip-10-0-130-5.ec2.internal\" DevicePath \"\""
Oct 30 01:41:04.595155 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:04.595126    4576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c"
Oct 30 01:41:04.595340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:04.595164    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-6ns7c" event={"ID":"e6fb3a61-756c-47af-ac43-7a078a22b19c","Type":"ContainerDied","Data":"fef3a9ce3c91e2c9e2b8154bfc0078d423846187a19eed64cf5be511ae3e7ba0"}
Oct 30 01:41:04.595340 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:04.595210    4576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fef3a9ce3c91e2c9e2b8154bfc0078d423846187a19eed64cf5be511ae3e7ba0"
Oct 30 01:41:11.850781 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:11.850746    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mxzkv"]
Oct 30 01:41:18.380890 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:41:18.380849    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:41:22.360649 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:22.360615    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hp8xp_61283cde-12bc-48f5-8bda-faa1da468ca7/manager/0.log"
Oct 30 01:41:22.553286 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:22.553249    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-gkxfc_8fadd188-7027-434e-9d8f-f9d63a62ef6b/manager/0.log"
Oct 30 01:41:22.757181 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:22.757154    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fnvkh_37c0bec0-ca1a-4194-bf8e-7c530a8b1aea/manager/0.log"
Oct 30 01:41:23.226653 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:23.226622    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x78xr_97c70240-814b-4be3-9c3e-3bc79a889b38/server/0.log"
Oct 30 01:41:23.415353 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:23.415321    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-zkgm7_0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f/manager/0.log"
Oct 30 01:41:23.625640 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:23.625532    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-6d7mf_d049c7b9-2cae-42e9-b404-fb375a91a2cf/manager/0.log"
Oct 30 01:41:23.851437 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:23.851407    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8cc7fc69d-7kw5f_2e93e8d8-9c70-446a-ad48-ead2f959237a/copy-manifests/0.log"
Oct 30 01:41:23.880539 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:23.880466    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8cc7fc69d-7kw5f_2e93e8d8-9c70-446a-ad48-ead2f959237a/manager/0.log"
Oct 30 01:41:25.632651 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:25.632618    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-nt2wm_33e689d7-6d8e-4433-8b0a-36fab7df563c/revoke-keys/0.log"
Oct 30 01:41:26.541947 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:26.541919    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-6ns7c_e6fb3a61-756c-47af-ac43-7a078a22b19c/revoke-keys/0.log"
Oct 30 01:41:27.065599 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:27.065569    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-rkzgr_ffd79150-1053-4599-9d34-b7af829a2937/revoke-keys/0.log"
Oct 30 01:41:28.912829 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:28.912795    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-jzffv_c7b2a498-5dc3-4d75-9859-f064b91e522f/manager/0.log"
Oct 30 01:41:29.710780 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:29.710745    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mxzkv_d6c4055c-973c-4e7f-80cf-2dae06fae287/limitador/0.log"
Oct 30 01:41:30.710657 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:30.710624    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:41:30.857019 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:30.856982    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:41:31.960413 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:31.960380    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-bbcwh_deb10f38-0a8d-428e-a7bf-8aa9d4392baf/istio-proxy/0.log"
Oct 30 01:41:32.353834 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:32.353755    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-mb4zz_4ee221b6-8c75-451b-b949-e90edd2185a6/istio-proxy/0.log"
Oct 30 01:41:32.941320 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:32.941287    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-x8j72_d434c59a-04fd-493c-a603-88326e67ae06/istio-proxy/0.log"
Oct 30 01:41:44.611673 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:44.611638    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:41:45.335588 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:45.335554    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hp8xp_61283cde-12bc-48f5-8bda-faa1da468ca7/manager/0.log"
Oct 30 01:41:45.537628 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:45.537591    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-gkxfc_8fadd188-7027-434e-9d8f-f9d63a62ef6b/manager/0.log"
Oct 30 01:41:45.745882 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:45.745855    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fnvkh_37c0bec0-ca1a-4194-bf8e-7c530a8b1aea/manager/0.log"
Oct 30 01:41:46.171989 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:46.171900    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x78xr_97c70240-814b-4be3-9c3e-3bc79a889b38/server/0.log"
Oct 30 01:41:46.375154 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:46.375122    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-zkgm7_0b9f28eb-aba2-47ed-a91b-43b7cf2ec71f/manager/0.log"
Oct 30 01:41:46.567596 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:46.567565    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-6d7mf_d049c7b9-2cae-42e9-b404-fb375a91a2cf/manager/0.log"
Oct 30 01:41:46.791317 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:46.791289    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8cc7fc69d-7kw5f_2e93e8d8-9c70-446a-ad48-ead2f959237a/copy-manifests/0.log"
Oct 30 01:41:46.821468 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:46.821373    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8cc7fc69d-7kw5f_2e93e8d8-9c70-446a-ad48-ead2f959237a/manager/0.log"
Oct 30 01:41:48.354410 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:48.354383    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-nt2wm_33e689d7-6d8e-4433-8b0a-36fab7df563c/revoke-keys/0.log"
Oct 30 01:41:48.380077 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:41:48.380044    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:41:49.239369 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:49.239338    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-6ns7c_e6fb3a61-756c-47af-ac43-7a078a22b19c/revoke-keys/0.log"
Oct 30 01:41:49.771007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:49.770970    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-rkzgr_ffd79150-1053-4599-9d34-b7af829a2937/revoke-keys/0.log"
Oct 30 01:41:51.212621 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:51.212573    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:41:51.236403 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:51.236376    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-jzffv_c7b2a498-5dc3-4d75-9859-f064b91e522f/manager/0.log"
Oct 30 01:41:52.056080 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:52.056051    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mxzkv_d6c4055c-973c-4e7f-80cf-2dae06fae287/limitador/0.log"
Oct 30 01:41:54.017719 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:54.017686    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-bbcwh_deb10f38-0a8d-428e-a7bf-8aa9d4392baf/istio-proxy/0.log"
Oct 30 01:41:54.380709 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:54.380624    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-mb4zz_4ee221b6-8c75-451b-b949-e90edd2185a6/istio-proxy/0.log"
Oct 30 01:41:54.998195 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:41:54.998162    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-x8j72_d434c59a-04fd-493c-a603-88326e67ae06/istio-proxy/0.log"
Oct 30 01:42:18.380773 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:42:18.380735    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:42:22.091763 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:42:22.091680    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:42:44.725978 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:42:44.725925    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:42:47.270827 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:42:47.270788    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:42:48.379881 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:42:48.379844    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:42:48.664079 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:42:48.663995    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:42:48.778578 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:42:48.778541    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:42:51.325292 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:42:51.325253    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:42:52.230938 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:42:52.230895    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:42:52.337094 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:42:52.337052    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:43:07.727524 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:07.727485    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:43:07.842441 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:07.842399    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:43:12.018646 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:12.018613    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4h2rk_56a648c9-5e85-49b8-91f3-967fad2836f8/global-pull-secret-syncer/0.log"
Oct 30 01:43:12.875879 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:12.875845    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log"
Oct 30 01:43:13.396143 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:13.396115    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-5.ec2.internal_f3b3e87c8faf211497319716dcdc8a4c/haproxy/0.log"
Oct 30 01:43:18.380716 ip-10-0-130-5 kubenswrapper[4576]: E1030 01:43:18.380676    4576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tqzf" containerName="extractor"
Oct 30 01:43:20.443052 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:20.443022    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-jzffv_c7b2a498-5dc3-4d75-9859-f064b91e522f/manager/0.log"
Oct 30 01:43:20.907843 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:20.907750    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mxzkv_d6c4055c-973c-4e7f-80cf-2dae06fae287/limitador/0.log"
Oct 30 01:43:22.299735 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.299697    4576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"]
Oct 30 01:43:22.300316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.300262    4576 cpu_manager.go:414] "containerMap: removing container" podUID="e6fb3a61-756c-47af-ac43-7a078a22b19c" containerName="revoke-keys"
Oct 30 01:43:22.300316 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.300282    4576 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fb3a61-756c-47af-ac43-7a078a22b19c" containerName="revoke-keys"
Oct 30 01:43:22.300472 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.300408    4576 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fb3a61-756c-47af-ac43-7a078a22b19c" containerName="revoke-keys"
Oct 30 01:43:22.303568 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.303545    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.320276 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.320223    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"]
Oct 30 01:43:22.332490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.332462    4576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dw966\"/\"default-dockercfg-rnrqh\""
Oct 30 01:43:22.332490 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.332491    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dw966\"/\"kube-root-ca.crt\""
Oct 30 01:43:22.332698 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.332547    4576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dw966\"/\"openshift-service-ca.crt\""
Oct 30 01:43:22.426907 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.426872    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-sys\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.427084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.426932    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-podres\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.427084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.427012    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552hg\" (UniqueName: \"kubernetes.io/projected/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-kube-api-access-552hg\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.427084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.427043    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-proc\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.427084 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.427079    4576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-lib-modules\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528545    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552hg\" (UniqueName: \"kubernetes.io/projected/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-kube-api-access-552hg\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528581 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528589    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-proc\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528622    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-lib-modules\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528651    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-sys\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528685    4576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-podres\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528743    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-proc\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528762    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-podres\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528750    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-sys\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.528809 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.528760    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-lib-modules\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.549663 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.549637    4576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552hg\" (UniqueName: \"kubernetes.io/projected/07b0fb60-f4cb-4fa0-af6d-c1fc89597def-kube-api-access-552hg\") pod \"perf-node-gather-daemonset-lmltp\" (UID: \"07b0fb60-f4cb-4fa0-af6d-c1fc89597def\") " pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.616594 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.616510    4576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:22.911404 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:22.911379    4576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"]
Oct 30 01:43:22.914994 ip-10-0-130-5 kubenswrapper[4576]: W1030 01:43:22.914961    4576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07b0fb60_f4cb_4fa0_af6d_c1fc89597def.slice/crio-e153ef85227b47add285dafa8dba82c26daa6abf4a309c5d3ae04ccb1f65836b WatchSource:0}: Error finding container e153ef85227b47add285dafa8dba82c26daa6abf4a309c5d3ae04ccb1f65836b: Status 404 returned error can't find the container with id e153ef85227b47add285dafa8dba82c26daa6abf4a309c5d3ae04ccb1f65836b
Oct 30 01:43:23.195575 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:23.195538    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp" event={"ID":"07b0fb60-f4cb-4fa0-af6d-c1fc89597def","Type":"ContainerStarted","Data":"aecea333e8adefe0ec9465ddfada3c94d072069b77d442fa2e5585b01476ce44"}
Oct 30 01:43:23.195792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:23.195598    4576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp" event={"ID":"07b0fb60-f4cb-4fa0-af6d-c1fc89597def","Type":"ContainerStarted","Data":"e153ef85227b47add285dafa8dba82c26daa6abf4a309c5d3ae04ccb1f65836b"}
Oct 30 01:43:23.195792 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:23.195682    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:23.236552 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:23.236499    4576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp" podStartSLOduration=1.236485856 podStartE2EDuration="1.236485856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:23.235662932 +0000 UTC m=+3883.306986767" watchObservedRunningTime="2026-10-30 01:43:23.236485856 +0000 UTC m=+3883.307809690"
Oct 30 01:43:23.707731 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:23.707707    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-8l77x_4219ea51-a9b2-4276-b929-ad28209dff06/monitoring-plugin/0.log"
Oct 30 01:43:24.153839 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:24.153761    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fllj8_ecfbfe46-06c9-488c-8016-72d31176fdd5/node-exporter/0.log"
Oct 30 01:43:24.252062 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:24.252033    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fllj8_ecfbfe46-06c9-488c-8016-72d31176fdd5/kube-rbac-proxy/0.log"
Oct 30 01:43:24.349841 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:24.349814    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fllj8_ecfbfe46-06c9-488c-8016-72d31176fdd5/init-textfile/0.log"
Oct 30 01:43:25.113451 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:25.113423    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0/prometheus/0.log"
Oct 30 01:43:25.212652 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:25.212622    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0/config-reloader/0.log"
Oct 30 01:43:25.311800 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:25.311765    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0/thanos-sidecar/0.log"
Oct 30 01:43:25.410730 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:25.410643    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0/kube-rbac-proxy-web/0.log"
Oct 30 01:43:25.509314 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:25.509287    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0/kube-rbac-proxy/0.log"
Oct 30 01:43:25.608612 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:25.608585    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0/kube-rbac-proxy-thanos/0.log"
Oct 30 01:43:25.710895 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:25.710868    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80fb8b2a-c3f2-4c42-bcfb-4a0d3f07fab0/init-config-reloader/0.log"
Oct 30 01:43:29.211573 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:29.211548    4576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dw966/perf-node-gather-daemonset-lmltp"
Oct 30 01:43:30.341135 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:30.341099    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-6l59b_a3e0ce63-11e7-447b-a015-eda0fa7adeab/download-server/0.log"
Oct 30 01:43:33.193048 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:33.193017    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:43:33.297859 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:33.297823    4576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62frw_d10c0ce2-a04c-4f4f-9977-db48be2e5504/konnectivity-agent/0.log" limit=1024
Oct 30 01:43:33.578382 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:33.578302    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k54rc_1b9f3661-225a-43e3-8f77-5122261f257f/dns/0.log"
Oct 30 01:43:33.675391 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:33.675365    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k54rc_1b9f3661-225a-43e3-8f77-5122261f257f/kube-rbac-proxy/0.log"
Oct 30 01:43:34.223007 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:34.222957    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6j4cc_edfab0af-8070-4ffb-9c66-9c94399393d7/dns-node-resolver/0.log"
Oct 30 01:43:35.421740 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:35.421708    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-646d6d98c-rp7xm_d7c9b38a-278c-43ef-92ba-9987367dbb13/registry/0.log"
Oct 30 01:43:35.654018 ip-10-0-130-5 kubenswrapper[4576]: I1030 01:43:35.653985    4576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-frssv_38381a16-212b-4d9c-98c2-6d81e4af3b74/node-ca/0.log"